Data Engineer Lead - Java with Spark

Data Engineer / Senior / Supervisor

Data Platform Engineer

Senior Data Engineer

Responsibilities:Lead and manage the development team, including assigning tasks and monitoring progressPart of Development team to analyse, design, and implement enterprise-wide Java based systemsHands on expertise in Microservices.Plan, analyse and define solutions through business, functional and technical expertiseWork with product managers on issues or questions related to system requirementsRespond to inquiries concerning application development and supportDevelop, program, and maintain applications using the Apache Spark open-source frameworkWork with different aspects of the Spark ecosystem, including Spark SQL, DataFrames, Datasets, and streamingRequirements:At least a bachelor’s degree in computer engineering, or a relevant fieldMinimum 10 years of hands-on experience in Spark, Microservices and latest Java versionMinimum 6 years of experience in cloud computing and architectureExperience with major cloud platforms e.g. AWS, Azure, GCP etc.Experience with containerization technologies e.g. Ec2, Docket, KubernetesExperience using Spark framework and streaming data platformsExperience in hands-on programming skills in Java and Hadoop ecosystemKnowledge in relational databases i.e. SQL and ORM technologies e.g. JPA2, HibernateKnowledge in database design and data normalisationKnowledge of DevOps methodologies and toolsKnowledge of Oracle / PostgreSQLKnowledge of relational databases, SQL and ORM technologies i.e. JPA2, Hibernate and database design and data normalizationPreferred Requirements:Knowledge and hands-on experience in Hadoop concepts like Hadoop Common, HDFS, Hadoop Yarn, Hadoop MapReduce and data processing techniquesPrevious work experiences as a Java, Spring boot Technical Lead or similar rolesStrong experience with Java and Spring boot frameworks like MVC, REST backend applications

Role Description:The Data Engineering Senior Supervisor will lead a small team of data engineering professionals to design, build and operate high performance data centric products or solutions through the use of modern engineering practices and technologies.In this role, you will provide guidance to ensure the delivered data centric products or solutions are scalable, sustainable and robust.You will also lead team development and cross team relationships and delivery to advance the company's engineering delivery.Responsibilities:Lead a local or regional team of engineering professionals that design, develop, deploy and enhance new and existing data centric products or solutions through the use of modern platforms and technologies.Provide routine inputs and guidance to the data engineering team on business requirements analysis, technical specification's preparation, product delivery, testing and operation to support businesses strategic direction.Coach engineers, share relevant technical approaches, identify new trends, modern skills and present new methodologies within the data engineering community.Lead and develop a team, coach and make decisions related to talent management, hiring, performance, and disciplinary actions.Other duties as assignedRequirements:Minimum QualificationsBachelor's degree in a related field or equivalent experience.Minimum of four years of related work experience.Minimum of two years leading a team of Data Engineers.Preferred QualificationsConfirmed data engineering experience with data storage and management of large, heterogenous datasets, including formats, structures, and cataloging with such tools as Iceberg, Parquet, Avro, ORC, S3, HFDS, HIVE, Kudu or others.Experience leading Big Data engineering teams working in environments such as Hadoop and SparkExperience leading technology strategies with cloud Platforms including AWS, GCP or AzureExperience leading engineering teams working in streaming and stream integration or middleware platforms, tools, and architectures such as Kafka, Flink, JMS, or Kinesis.Confirmed knowledge of best practice patterns and how to apply them to modern data architectures, including data warehouses, data lakes, data mesh, hubs and associated capabilities including ingestion, governance, modeling, observability and more.Experience leveraging transformation and modeling tools, including SQL based transformation frameworks, orchestration and quality frameworks including dbt, Apache Nifi, Talend, AWS Glue, Airflow, Dagster, Great Expectations, Oozie and others.Strong programming knowledge of SQL, Python, R, Java, Scala or equivalent.Proficiency in engineering tooling including docker, git, and container orchestration services.Strong experience leading devops teams with demonstratable understanding of associated best practices for code management, continuous integration, and deployment strategies.Experience and knowledge of data governance considerations including quality, privacy, security associated implications for data product development and consumption.A passion for quality and continuous improvement with an owners mindset.Ability to work effectively as part of a team, group and cultureAbility to navigate ambiguity and work in agile ways.

Responsibilities:Support daily business-as-usual (BAU) operations, audit and complianceCommunicate with vendor and BAU teams for data ingestion queries and issuesManage data ingestion and DAG maintenance in AirflowDevelop and modify Python scripts for data ingestionWrite optimised SQL queries in SnowflakeEnsure documentation of applicationsRequirements:Bachelor's degree in information technology/ computer science / software engineeringKnowledge and hands-on experience writing effective SQL queries and statementsUnderstanding of AWS servicesAt least 5 years of experience in a similar capacityAt least 3 years of proficiency in using Python to develop and modify scriptsAt least 3 years of proficiency in managing data ingestion and DAG maintenance in AirflowPreferred Requirements:Knowledge in Hadoop Ecosystem like Spark or PySparkKnowledge in AWS services like S3, Data Lake, Redshift, EMR, EC2, Lambda, Glue, Aurora, RDS, Airflow1-2 years of proficiency in using Snowflake, including query optimisation to control Snowflake’s warehouse costPrevious work experiences from investment banking/finance industry with demonstrated exposure to be ready to take on an individual contributor role

Responsibilities:Data engineers need to write code to develop, maintain, and optimize data pipelines, as well as to implement data storage solutions and data processing systems. They typically use programming languages such as Python, Java, or Scala, along with big data technologies like Hadoop, Spark, and KafkaDesigning scalable data architecture that supports both transactional and analytical workloads. This involves understanding business requirements and translating them into technical solutions.Developing robust data pipelines to ingest, cleanse, transform, and aggregate data from various sources (internal and external). This includes both batch and real-time/streaming data processing.Integrating data from disparate sources and ensuring data consistency, integrity, and reliability across different systems.Requirements:5-8 years experience with GCP, Big Query, Data Fusion, Analytics Hub and DataFlowExperience leading othersExperience managing projectsExcellent written and verbal communication skillsTime ManagementAbility to work alone and with a teamAbility to mentor junior membersA Bachelor’s degree in MIS or mathematics, statistics, or computer science, international equivalent, or equivalent job experience

Skills

Industry

Location

Jobtitle

4 hadoop Jobs found

Data Engineer Lead - Java with Spark

Data Engineer / Senior / Supervisor

Data Platform Engineer

Senior Data Engineer

WhiteCrow
WhiteCrow is on a mission to connect companies with the best talent across the globe.

EMPLOYER

TALENT

COMPANY

Contact Us

JOBS BY SKILLS

JOBS BY FUNCTION

JOBS BY INDUSTRY

JOBS BY LOCATIONS

MODEL JOBS

JOBS BY CITY

Skills

Industry

Location

Jobtitle

4 hadoop Jobs found

Data Engineer Lead - Java with Spark

Data Engineer / Senior / Supervisor

Data Platform Engineer

Senior Data Engineer

WhiteCrow WhiteCrow is on a mission to connect companies with the best talent across the globe.

EMPLOYER

TALENT

COMPANY

Contact Us

JOBS BY SKILLS

JOBS BY FUNCTION

JOBS BY INDUSTRY

JOBS BY LOCATIONS

MODEL JOBS

JOBS BY CITY

WhiteCrow
WhiteCrow is on a mission to connect companies with the best talent across the globe.