DATA ENGINEER - AZURE DATABRICKS & PYSPARK
Posted Updated
Description
Duncan & Ross is hiring a Data Engineer for a leading client in Dubai. You will design, build and run scalable data pipelines and data warehouse solutions on Azure and Databricks, turning raw data into trusted analytics-ready data.
3-5+ years of hands-on data engineering experience is required. You will work with Python/PySpark, SQL, and cloud services to deliver robust data products, while ensuring data quality and governance across heterogeneous sources.
Design, build and maintain ETL/ELT pipelines ingesting data from multiple sources into Azure and Databricks. Develop data transformations in Python and PySpark, optimising Spark jobs for performance and cost. Write efficient SQL for data modelling and analysis across databases. Build and manage data warehouses and lakehouses on Azure and Databricks, including Delta Lake architectures. Orchestrate data workflows with ADF, Databricks Workflows or Airflow, with monitoring and alerts. Implement data quality checks, validation rules, and reconciliation processes. Apply governance with cataloguing, lineage, access control and security (Unity Catalog / Purview). Collaborate with data analysts/scientists and stakeholders to deliver data products. Troubleshoot failures and performance issues; continuously improve reliability. Document pipelines, models and processes; follow CI/CD and version control. 3-5+ years hands-on data engineering experience. Data pipelines: ETL/ELT pipelines for batch/incremental loads. SQL & relational databases: advanced SQL with SQL Server/Azure SQL/PostgreSQL/Oracle. Big data: hands-on Apache Spark, performance tuning for large datasets. Cloud: strong Azure (Data Lake Storage, Data Factory, Synapse) & Databricks experience. Data warehousing/lakehouse design on Azure & Databricks (dimensional modelling; Delta Lake). Programming: strong Python and PySpark skills. Orchestration: Azure Data Factory, Databricks Workflows, Airflow or similar. Data quality, lineage, metadata, access controls; governance.Skills
- Airflow
- Analytics
- Azure
- Azure Data Factory
- Big Data
- CI/CD
- Cloud
- Data Engineering
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- Databricks
- Delta Lake
- Dimensional Modeling
- ELT
- ETL
- Lakehouse
- Oracle
- PostgreSQL
- PySpark
- Python
- RDBMS
- Spark
- SQL
- SQL Server
- Unity
- Unity Catalog
- Version Control
Sign in to read and join the discussion.