Data Engineer
Summary
Build and deploy cloud-native data platforms using Python, Spark, Kafka, and AWS, focusing on batch and streaming pipelines with modern DevOps practices.
This role is for a Modern Data Engineer with strong software engineering skills who can build and deploy cloud-native data platforms.
Key Requirements
Software Engineering
- Strong Python development skills (primary programming language)
- Solid understanding of software engineering principles and clean coding practices
- Experience working within the Software Development Life Cycle (SDLC)
- Proficient with Git and version control
- Experience with CI/CD pipelines and modern DevOps practices
- Ability to solve problems through code, not just SQL
- Hands-on experience with modern data engineering technologies, including:
- Spark / PySpark
- Databricks
- Kafka
- Flink
- Data Lake architectures
- Experience building both batch and streaming data solutions
Practical experience delivering solutions on at least one major cloud platform:
- Amazon Web Services (AWS)
Relevant services may include:
- Data processing and analytics tools (BigQuery, Synapse, Redshift, Databricks, Dataflow, Glue, EMR, Dataproc)
- Event streaming platforms (Pub/Sub, Event Hubs, Kinesis)
- Container platforms (GKE, AKS, EKS)
DevOps & Deployment
Experience with:
- Docker
- Infrastructure as Code (Terraform or similar)
- Monitoring and observability tools
Experience should be gained through real production projects, not certifications alone.
Additional Valuable Skills
- Data modelling (relational and dimensional)
- Data warehousing concepts
- Modern orchestration and data platforms such as Databricks, Kafka and Microsoft Fabric
Ideal Candidate
We're looking for a modern data engineer with a software engineering mindset. Candidates should have proven experience delivering cloud-based data solutions using Python, Spark, CI/CD, Git and modern data platforms. Experience limited to traditional SQL, SSIS or ETL development without modern engineering practices is unlikely to meet current client requirements.
What We Value
- Continuous learning and professional development
- Ability to contribute immediately within a modern engineering environment
- Client-facing consulting experience is advantageous