Intermediate Data Engineer
Summary
Builds and supports scalable, secure batch and real-time data pipelines for a technology consultancy in Johannesburg, integrating diverse sources into data marts and AWS cloud feeds. Core stack includes Python, PySpark/Spark, Talend, Hadoop/EMR and AWS, working hybrid (three days a week in the office).
We’re working with an established technology consultancy, and they’re looking for an Intermediate Data Engineer to build and support scalable, repeatable and secure data pipelines as part of an agile team.
You’ll build the infrastructure that turns raw data into insight — handling large volumes in batch and real time, integrating diverse sources, and using modern big-data and cloud technologies.
What you’ll do
- Build and support scalable, secure data pipelines, integrating diverse data sources
- Handle large volumes of data in batch and real time
- Create data marts using Talend or a similar ETL tool, and manipulate data using Python
- Process data using the Hadoop paradigm (e.g. EMR), and create data feeds into AWS Cloud
- Develop for big data and business intelligence, including automated testing and deployment
- Translate functional and technical requirements into performant solutions
What you’ll bring
- 3–5 years’ data-engineering or software-engineering experience, including big-data and ETL work
- Strong Python, and PySpark or Spark
- Big-data batch and streaming tools, and AWS (EMR, EC2, S3)
- Data modelling and data-warehousing experience
- A bachelor’s degree in Computer Science, Computer Engineering or equivalent
Advantageous
- AWS certification
- BigQuery, DBT and a combination of AWS / GCP / Azure
- AI tooling (e.g. Bedrock or similar)
Working model
Hybrid in Johannesburg — three days a week in the office.
#J-18808-Ljbffr