Senior Data Engineer (AWS & Databricks) | Contract

Key Responsibilities

• Design, develop and maintain scalable batch and streaming data pipelines across AWS and Databricks.
• Build and manage lakehouse architectures using Apache Spark, PySpark and Delta Lake, including Bronze, Silver and Gold data layers.
• Integrate data from databases, APIs, files and event streams using incremental ingestion and change data capture.
• Implement data quality checks, schema validation, reconciliation, metadata management and data lineage.
• Develop automated workflows and CI/CD pipelines using Databricks Workflows, AWS services, Git and infrastructure-as-code tools.
• Monitor pipeline performance, troubleshoot issues and implement logging, alerts and recovery procedures.
• Optimise Spark and SQL workloads for performance, reliability and cloud cost efficiency.
• Implement data security, access controls, encryption and governance in line with agency requirements.
• Establish engineering standards, review code and architecture, and mentor junior engineers.
• Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.
• Maintain technical documentation, operating procedures and knowledge-transfer materials.

Requirements & Qualifications

• Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related field.
• 6-10 years of relevant data or software engineering experience, including at least 2 years working with cloud-based data platforms.
• Strong hands-on experience in Python, PySpark, SQL, Apache Spark and Delta Lake.
• Proven experience building and operating production data pipelines using Databricks and AWS.
• Good knowledge of AWS services such as S3, Glue, Lake Formation, IAM, Lambda, Redshift and CloudWatch.
• Experience with lakehouse architecture, data modelling, incremental processing, change data capture and data quality management.
• Familiarity with workflow orchestration, CI/CD, Git and infrastructure-as-code tools such as Terraform.
• Experience with pipeline monitoring, troubleshooting, performance optimisation and cloud security.
• Ability to lead technical design, conduct code reviews, establish engineering standards and mentor engineers.
• Strong problem-solving, communication and stakeholder management skills.

Preferred Skills

• Experience with Unity Catalog, Databricks Workflows, Lakeflow, Delta Live Tables or Structured Streaming.
• Familiarity with Airflow, dbt, Kafka, AWS DMS, MSK or Kinesis.
• Experience building data products, data contracts, feature engineering or ML data pipelines.
• Knowledge of Singapore public-sector data governance, security and compliance requirements.
• Relevant Databricks or AWS certifications.