Data Engineer (SSE / Staff Engineer) - Python & Spark & AWS
Summary
Design and build scalable data pipelines and analytics infrastructure using Python, Spark, and AWS for large-scale data processing and real-time streaming.
We are seeking a highly skilled Senior Software Engineer (SSE) / Staff Data Engineer to design, build, and optimize large-scale data solutions. The ideal candidate will have strong expertise in Python, Spark, and AWS, with a proven track record in developing robust data pipelines, enabling advanced analytics, and ensuring data governance and security.
Key Responsibilities
- Design and implement scalable ETL pipelines for data ingestion, transformation, and integration.
- Ensure data quality, governance, and compliance across all stages.
- Develop and optimize solutions using Apache Spark, Hadoop, Hive, and Kafka.
- Handle large-scale data processing and real-time streaming.
- Build and maintain data solutions on AWS (S3, Glue, Redshift, Athena, IAM).
- Implement secure and cost-efficient cloud architectures.
- APIs & Automation
- Develop RESTful APIs and automation scripts for data services and integrations.
- Collaborate with application teams to enable data-driven microservices.
- DevOps & CI/CD
- Implement CI/CD pipelines using Jenkins, Git, Docker, and Kubernetes.
- Ensure smooth deployment and monitoring of data applications.
- Security & Compliance
- Apply best practices for data governance, risk management, and regulatory compliance.
- Work closely with cross-functional squads and stakeholders to deliver data solutions in an Agile environment.
Required Skills & Experience
- Strong programming skills in Python (Scala experience is a plus).
- Hands‑on experience with Spark, Hadoop, Hive, and Kafka.
- Expertise in AWS services (S3, Glue, Redshift, Athena, IAM).
- Solid understanding of ETL processes, data modeling, and pipeline orchestration.
- Familiarity with RESTful APIs, microservices, and automation scripting.
- Knowledge of CI/CD tools, containerization (Docker), and orchestration (Kubernetes).
- Understanding of data security, compliance, and governance principles.
- Excellent problem‑solving skills and ability to work in a fast‑paced Agile environment.
Preferred Qualifications
- Experience with data lake and data warehouse architectures.
- Exposure to machine learning pipelines or advanced analytics.
- Certification in AWS or Big Data technologies.
Seniority level
Mid‑Senior level
Employment type
Contract
Job function
Information Technology
Industries
Software Development
Location: Sydney, New South Wales, Australia
Salary: A$140,000 - A$180,000