Apache Data Engineer (SC Cleared)
Posted 1
view
Summary
Build and maintain ETL data pipelines into a Lakehouse architecture across AWS and on-premises, using change data capture and event-driven ingestion, data modeling, and Git-based CI/CD within a sprint team. Core stack is Apache tooling plus Java/Groovy; UK SC security clearance is required.
- Design and build data pipelines and ETL processes into a Lakehouse architecture across cloud (AWS) and on premises platforms
- Implement change data capture and event-driven patterns to ingest data from source systems
- Develop data models for the pipeline layer
- Handle non-functional requirements including security, performance and metadata labelling
- Write unit and integration tests for pipeline components
- Work within a defined SDLC using Git-based CI/CD tooling
- Contribute to sprint delivery as part of an integrated development team
Experience building data pipelines within Lakehouse architectures - Comfortable with Apache tooling, for data engineering work
- Experience with event sourcing, change data capture and event-driven architectures
- Java and/or Groovy, with a solid grasp of object-oriented design principles
- Experience working in an SDLC with an integrated team on CI/CD tooling (Git)
Nice-to-have
- Apache NiFi
- AWS, particularly EMR, S3 and SQS
- Awareness of domain-driven design
- Linux and Scripting
- Postgres or PL/SQL