Senior Data Engineer (Databricks)

Primary Purposes:: We are looking for an experienced

Data Engineer

to join the delivery team for the

EDW Reporting Modernisation project . You will be responsible for designing and building the

Gold layer Data Mart tables, transformation pipelines, and associated views

on the

Unified Data Platform (Databricks) , working closely with the Tech Lead and Data Modeler to deliver a robust and scalable reporting data layer that caters to

Power BI reports . As the onshore Data Engineer,

you will serve as the

technical bridge

between the Singapore-based design team and the offshore engineering squad, providing

day-to-day technical direction, code review, and quality oversight

across the full Data Mart build.

Responsibilities:: Design and Development:: • Develop

Gold layer Data Mart tables

and associated database views in Databricks in accordance with signed-off Source-to-Target Mapping (STTM) documents and design specifications • Implement

Silver-to-Gold transformation pipelines

using Databricks Workflows, Delta Live Tables (DLT), and PySpark • Create

Delta table DDL in Unity Catalog , including schema definition, table properties, and column-level comments • Implement

SCD Type 2 merge logic

for dimension tables and aggregation logic for fact tables per the grain specification • Configure table partitioning,

Z-ORDER clustering, and Liquid Clustering

strategies for query performance optimisation • Develop and implement

data quality expectations

using Delta Live Tables DQ framework per the DQ thresholds defined in the design specification • Schedule and configure

Databricks Workflow job DAGs

for all pipeline runs, including watermark-based incremental load logic Testing and Quality Assurance:: • Conduct

unit testing

for each Data Mart table upon build completion — covering row count validation, null checks, duplicate grain checks, and transformation logic verification • Execute

system integration test (SIT) scenarios

for all 60 Data Mart tables, including aggregated value checks, referential integrity validation, and SCD integrity checks • Log, investigate, and resolve defects identified during

SIT and UAT , performing root cause analysis and documenting fixes • Review and validate unit test outputs from Pune Data Engineers before test results are submitted for Tech Lead review Offshore Squad Technical Leadership:: • Provide

day-to-day technical direction and coding guidance

to the 2 offshore-based Data Engineers • Conduct

code reviews

for all pipeline and DDL code produced by the offshore squad before submission for Tech Lead sign-off • Ensure offshore squad adherence to agreed

coding standards, naming conventions, Unity Catalog governance rules, and development best practices • Facilitate

knowledge sharing and technical problem-solving

with the Pune squad on complex transformation and merge logic Deployment and Handover:: • Execute

Data Mart pipeline and table deployment

to the production Databricks environment per the approved deployment runbook • Validate the

first production pipeline run

and confirm data freshness post-deployment • Contribute to the

operational runbook

covering pipeline architecture, job schedules, alert thresholds, and common failure scenarios • Support

knowledge transfer sessions

with the Singtel IT operations team during the handover phase

Qualifications Required Skills and Experience Technical — Mandatory •

5+ years of experience

in data engineering with a strong focus on ETL/ELT pipeline development and dimensional data modelling • Hands-on experience with

Databricks

— including

Delta Lake, Delta Live Tables, Databricks Workflows, Unity Catalog, and Databricks SQL • Proficiency in

PySpark and Databricks SQL

for large-scale data transformation • Strong understanding of

dimensional modelling concepts

— star schema, SCD Type 2, surrogate key design, fact and dimension table design • Experience implementing

data quality frameworks and reconciliation testing • Familiarity with

Source-to-Target Mapping (STTM)

and translating design specifications into production-grade pipelines • Experience with

incremental load patterns

— watermark-based, partition-based, or CDC-driven • Proficiency in

Git-based version control

for collaborative development Technical — Preferred:: • Experience with

Databricks Unity Catalog access control, table tagging, and lineage tracking • Exposure to

Oracle-to-Databricks migration projects

or similar platform modernisation programmes • Familiarity with

Power BI Semantic Models

and how Gold layer table design impacts downstream DAX measure performance • Experience with

Databricks Assistant or AI-assisted code generation tools

for accelerated pipeline development • Knowledge of

CI/CD pipeline setup

for Databricks notebook or YAML