Data Engineer
Summary
Build and maintain Spark pipelines and Kafka ingestion for large-scale government data lakes, ensuring data quality and real-time processing.
Data Engineer – Spark & Kafka
We are looking for a Data Engineer with experience in Spark and Kafka to operate with one of our large Federal Government Clients. As a Data Engineer, you will have the opportunity to work with cutting‑edge technologies and collaborate with cross‑functional teams to ensure successful data integration.
Responsibilities
- Maintain and develop Spark pipelines to process large‑scale data sets.
- Ingest tables into our data lake from Kafka using batch processing, micro‑batching, or real‑time processing techniques.
- Run Spark jobs for ingestion, curation, and extraction of data.
- Alter schemas and configs to meet business requirements and ensure data quality.
- Attend meetings regarding data ingestion and collaborate with cross‑functional teams to ensure successful data integration.
- Troubleshoot and resolve bugs in the data processing pipelines.
- Monitor performance and troubleshoot issues using logging and monitoring tools.
- Document best practices and develop standards for data processing pipelines.
- Stay up‑to‑date with emerging technologies and industry trends in big data processing and analytics.
Requirements
- Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related field.
- 3+ years of experience in data engineering, software development, or a related field.
- Strong programming skills in Scala, Python, or Java.
- Expertise in Spark, Kafka, and big data processing concepts.
- Experience with data storage solutions such as Hadoop HDFS or Amazon S3, and data serialization formats.
- Knowledge of database concepts, data modelling, schema design, and normalisation.
- Familiarity with monitoring and logging tools, and workflow management tools such as Apache Airflow.
- Experience with version control systems such as Git.
- Security clearance required – Baseline (NV1 preferred).
Attributes and Aptitude
- Strong communication, problem‑solving, and collaboration abilities.
- Ability to work independently and in a team‑oriented, collaborative environment.
Benefits
We offer competitive compensation packages and benefits, along with opportunities for growth and advancement within the company. As a member of our team, you will have the chance to work on challenging projects, develop your skills, and make a real impact on our business.