Senior Data Engineer - #1732

Key Responsibilities
• Data Pipeline Architecture and Delivery: Lead the design and implementation of maintainable batch and streaming pipelines that ingest, validate, transform and publish structured, semi-structured and unstructured data across AWS and Databricks.
• Lakehouse and Data Product Engineering: Design Bronze, Silver and Gold data layers using Delta Lake or equivalent patterns; develop reusable domain data models, curated datasets and data products with explicit ownership, definitions and service expectations.
• Source Integration and Incremental Processing: Implement reliable ingestion from databases, files, APIs and event streams using full-load, incremental, change-data-capture and replayable processing patterns appropriate to each source.
• Data Quality and Contracts: Establish schema enforcement, data contracts, automated validation, reconciliation and anomaly checks covering completeness, validity, consistency, uniqueness, timeliness and referential integrity.
• Orchestration and Deployment Automation: Build parameterised workflows using Databricks Workflows or Lakeflow and relevant AWS services; integrate automated testing, nfrastructure as code and CI/CD controls
across development, test and production environments.
• Metadata, Lineage and Governance: Implement technical metadata, business definitions, end-to-end lineage, classification, ownership and access controls using Unity Catalog, AWS Glue Data Catalog, Lake Formation or approved platform capabilities.
• Pipeline Monitoring and Reliability: Define dashboards, alerts, logging, operational metrics and runbooks for pipeline health, freshness, data-quality failures, latency, throughput and downstream impact; lead troubleshooting and root-cause analysis.
• Performance and Cost Optimisation: Optimise Spark and SQL workloads, file sizes, partitioning, clustering,
caching, compaction, cluster configuration and workload scheduling to improve runtime, reliability and AWS/Databricks cost efficiency.
• Security and Data Protection: Apply least-privilege IAM, encryption, secrets management, environment separation, audit logging, sensitive-data controls and secure data-sharing patterns in line with applicable agency requirements.
• Technical Leadership and Quality: Set engineering standards, review designs and code, mentor engineers, manage technical debt and promote reusable components rather than project-specific pipelines.
• Stakeholder Collaboration: Work with data owners, stewards, analysts, data scientists, architects, product
teams and governance stakeholders to translate analytical and operational needs into implementable data specifications.
• Documentation and Knowledge Transfer: Maintain architecture decisions, source-to-target mappings, data models, contracts, operating procedures and support documentation, and transfer knowledge to the internal team throughout the engagement.

Requirements & Qualifications
A. Educational Qualification & Experience
Education: Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related technical discipline.
Experience: Typically 7–10 years of relevant data or software engineering experience, including hands-on design and
operation of production data pipelines and at least 2 years working with cloud-based data platforms. Candidates should demonstrate technical leadership through architecture, engineering standards, code review, mentoring or ownership of critical data services.

B. Technical & Professional Competencies
Essential (Must-Have)
• Strong Python, PySpark and SQL capability, including distributed processing, complex transformations, query optimisation, debugging and automated testing.
• Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming data engineering.
• Hands-on AWS data engineering experience with a relevant mix of S3, Glue, Lake Formation, IAM, KMS, Lambda, Step Functions, EventBridge, CloudWatch, Redshift and MSK/Kinesis.
• Experience designing lakehouse or data-lake architectures, including layered data patterns, dimensional or domain data modelling and reusable curated datasets.
• Experience implementing incremental ingestion, change data capture, idempotency, late-arriving data handling, backfills, replay and recovery.
• Experience implementing data quality, schema validation, reconciliation, metadata and lineage as automated parts of production pipelines.
• Experience with workflow orchestration, Git-based development, CI/CD, infrastructure as code such as Terraform, and environment promotion controls.
• Ability to implement pipeline observability, operational dashboards, alerting, runbooks, incident diagnosis and service recovery.
• Working knowledge of cloud security, least-privilege access, network boundaries, encryption, secret management, data classification and audit logging.
• Ability to review technical designs and code, establish engineering standards, and guide less-experienced engineers.

Preferred (Advantageous)
• Experience with Databricks Unity Catalog, Lakeflow or Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles.
• Experience with Apache Airflow, dbt, Kafka, Amazon MSK/Kinesis, AWS DMS or equivalent ingestion and orchestration technologies.
• Experience designing data products, data contracts, semantic or context layers, master/reference data patterns or data mesh-aligned operating models.
• Experience supporting feature engineering, ML training datasets, feature stores or MLOps pipelines on Databricks and AWS.
• Experience in Singapore public-sector environments, including IM8-aligned controls, data classification, VAPT,
system assurance or governed cross-agency data sharing.
• Relevant certifications such as Databricks Certified Data Engineer Professional, AWS Certified Data Engineer –
Associate or AWS Certified Solutions Architect – Associate/Professional.

C. Soft Skills & Communication Attributes
• Strong analytical and systems-thinking capability, with the discipline to identify failure modes, edge cases and
operational dependencies before production deployment.
• Clear written and verbal communication, including the ability to explain data architecture, quality, lineage,
security, cost and delivery trade-offs to technical and management audiences.
• Collaborative working style across data engineering, platform engineering, analytics, data science, architecture,
cybersecurity and governance teams.
• Ability to translate ambiguous business needs into clear data definitions, contracts, acceptance criteria and
incremental delivery plans.
• Strong ownership and production mindset, including careful documentation, support readiness and follow-through on incidents and technical debt.
• Ability to mentor engineers constructively, raise engineering standards and build sustainable in-house capability rather than creating vendor dependency.