Data Engineer
Summary
Designs and builds scalable AWS data pipelines using Python, PySpark, Airflow, and streaming tech to process high-volume data for analytics and warehousing.
My client is seeking a Senior Data Engineer with strong AWS data engineering experience to design, build and optimise scalable data pipelines across both batch and real-time environments. This role is ideal for someone with expertise in Python, PySpark, Airflow and AWS services, who enjoys working on modern cloud data platforms and high-volume data processing.
Key Responsibilities
- Design, develop and maintain scalable batch and real-time data ingestion frameworks.
- Build and optimise data pipelines using Python, PySpark, SQL and AWS services.
- Develop streaming solutions using AWS Kinesis, Kafka or similar technologies.
- Build and manage workflows using Apache Airflow (MWAA).
- Work with Amazon Redshift, RDS, S3 and NoSQL databases to support data warehousing and analytics.
- Implement data transformation, monitoring and observability solutions.
- Collaborate with cross-functional teams to deliver secure, high-performing data solutions.
Must Have Skills
- Strong commercial experience with AWS data services.
- Advanced Python, PySpark and SQL skills.
- Experience with Apache Airflow (MWAA).
- Hands‑on experience with Amazon Redshift.
- Experience building batch and real-time data pipelines.
- Strong knowledge of AWS Kinesis, Kafka or other streaming technologies.
- Experience working with Data Warehouses and Data Lake/Lakehouse architectures.
- Experience with relational and NoSQL databases such as Oracle, Snowflake, MongoDB, DynamoDB or Teradata.
Nice to Have
- Experience with AWS SageMaker Unified Studio.
- Exposure to ML or Generative AI pipelines.
- Data modelling experience.
- Infrastructure as Code using Terraform or CloudFormation.
- CI/CD and Git experience.
Paxus values diversity and welcomes applications from Indigenous Australians, people from diverse cultural and linguistic backgrounds and people living with a disability.