Senior Data Engineer
Summary
Build and optimize Databricks pipelines using Delta Live Tables and PySpark, migrate legacy systems to modern CI/CD, and ensure output equivalence with Palantir Foundry.
Senior Data Engineer - UAE
Position Overview
Location: Abu Dhabi, UAE
Work Requirement: Client requirement to have people onsite with office visits.
Key Responsibilities
- Implementing Databricks pipelines using Delta Live Tables (declarative pipelines).
- Rewriting proprietary Foundry transforms into optimised PySpark on Databricks.
- Moving from Foundry's build system to modern CI/CD (IaC-managed Databricks workspaces).
- Performing performance tuning on Databricks Photon to match or exceed Foundry interactivity.
- Contributing to the reconciliation engine (Palantir vs Databricks output equivalence).
- Managing pipeline rebuild, validation, internal testing, and UAT preparation.
Required Skills & Experience
- Proven production-level Databricks experience.
- Strong hands-on coding capability.
- Experience building and optimizing modern data pipelines.
- Ability to work within pod-based delivery teams.
- Experience owning pipeline migration and rebuild activities from design through validation.
- Databricks Databricks hands-on experience.
- Delta Live Tables.
- Unity Catalog.
- Workflows.
- Production experience required (not lab-based experience).
- Data Engineering & Development PySpark.
- Advanced SQL.
- Data modelling.
- Schema design.
- Medallion architecture.
- Lakehouse architecture.
- DevOps & Deployment Modern CI/CD for data pipelines.
- Infrastructure as Code (IaC).
- Git-based deployment.
- Communication Strong written and verbal English skills.
- Ability to interact directly with Databricks Professional Services (DBX).
Preferred Qualifications
- Certifications Databricks Certified Data Engineer Professional.
- Associate certification at minimum.
- Domain Experience Financial services.
- Audit.
- Asset management.
- Platform Experience Prior Palantir Foundry exposure, including: Code Repo.
- Code Workbook.
- Pipeline Builder.
- Migration Experience.
- Experience with migrations from legacy platforms such as: Informatica.
- Hyperion.
- Oracle.
- Palantir Foundry.
- Databricks.
Skills
- Candidates with any of the following experience will be advantageous: Azure Data Factory (ADF).
- Azure Synapse.
- dbt.
- Airflow.
- Equivalent orchestration tools.
Note: Client estate is Azure-heavy.