Azure Data Engineer (6-Month) - UAE, Data Lake & Pipelines
Posted Updated
Description
Datamatics Technologies LLC in UAE is seeking a Data Engineer to design and operate Azure-based data pipelines feeding lakehouse architectures. The role collaborates with data architects, QA engineers and analysts to deliver production-grade pipelines with governance, security, and lineage.
You will build batch, streaming and CDC ingestions using Azure Data Factory and Fabric, implement SCDs, data marts, and metadata capture in Purview, while adhering to local regulations (PDPL/DESC ISR).
Build batch, incremental, CDC and streaming ingestion pipelines from source systems into the Bronze layer, using Azure Data Factory and Fabric pipelines. Develop Silver- and Gold-layer transformations in SQL, PySpark and Spark notebooks, including cleansing, conformance, enrichment and business-rule logic. Implement dimensional models, SCDs, surrogate keys and data marts according to the architect's designs. Integrate DQ checks, validation, quarantine and reconciliation into pipelines in partnership with the Senior Data Quality Engineer. Ensure that pipelines publish metadata and lineage to Microsoft Purview. Implement scheduling, dependency management, parameterization, error handling and alerting. Optimize pipelines, partitioning, file formats (Delta/Parquet) and compute usage for performance and cost efficiency. Apply access controls, masking, encryption and secure credential management in line with UAE PDPL, the Dubai Data Law and DESC ISR. Maintain code in source control and deploy through CI/CD pipelines, with automated tests and environment promotion. Support production operations, incident resolution and runbooks, and coach counterparts so that the pipelines can be sustained after handover. 4+ years in data engineering, preferably within UAE government or regulated-sector environments. Advanced SQL, Python and PySpark/Spark skills. Hands-on experience with Azure Data Factory, Microsoft Fabric, Databricks or Synapse. Experience with CI/CD (Azure DevOps or GitHub), source control and DataOps practices. Familiarity with Microsoft Purview metadata and lineage, and with DQ tooling. Awareness of UAE data regulation, including UAE PDPL, the Dubai Data Law and DESC ISR. Excellent English communication and documentation skills.Skills
Sign in to read and join the discussion.