Senior Data Engineer - Azure Databricks, DBT & Capital Markets

Job Description: Senior Data Engineer Azure Databricks, dbt & Capital Markets

Overview

We are looking for a highly experienced Senior Data Engineer with 10+ years of experience in Data Engineering, specializing in Azure Databricks, dbt (Data Build Tool), and Capital Markets. The ideal candidate should have strong hands-on experience designing and developing scalable data platforms, modern ELT pipelines, and data transformation solutions, preferably within the Capital Markets or Financial Services domain.

Key Responsibilities

1. Design, develop, and optimize scalable data engineering solutions using Azure Databricks.
2. Build and maintain robust ELT pipelines using dbt (Data Build Tool).
3. Develop complex data transformations using PySpark, Spark SQL, and advanced SQL.
4. Implement Medallion Architecture (Bronze, Silver, and Gold) using Delta Lake.
5. Develop and manage dbt models, macros, snapshots, incremental models, and automated testing frameworks.
6. Design and implement data pipelines for Capital Markets data, including trade, transaction, market data, pricing, and reference data.
7. Work with large and complex financial datasets, ensuring data accuracy, consistency, and performance.
8. Implement data quality checks, reconciliation, validation, and data governance practices.
9. Optimize Databricks workloads, Spark jobs, SQL queries, and dbt transformations.
10. Implement CI/CD pipelines and version control using Git, GitHub, and Azure DevOps.
11. Collaborate with solution architects, business stakeholders, and cross-functional teams to deliver enterprise-grade data solutions.

Required Qualifications & Skills

• 10+ years of overall experience in Data Engineering.
• Strong hands-on experience with Azure Databricks and dbt.
• Excellent proficiency in PySpark, Spark SQL, and advanced SQL.
• Extensive experience in Delta Lake and Medallion Architecture.
• Strong experience developing dbt models, macros, snapshots, incremental models, and tests.
• Good understanding of modern ELT architecture and large-scale data processing.
• Strong experience with data modeling, Star Schema, and Snowflake Schema.
• Hands-on experience with Git, GitHub, CI/CD, and deployment practices.
• Strong understanding of data quality, data governance, and performance optimization.

Capital Markets Domain Experience

• Strong exposure to Capital Markets, Investment Banking, or Financial Services.
• Understanding of financial market data, trade lifecycle, transactions, securities, and pricing data.
• Experience handling market data, reference data, trade data, and portfolio-related datasets.
• Knowledge of financial data processing, reconciliation, regulatory reporting, and risk analytics is preferred.

Preferred Skills

• Experience with Azure Data Factory or Apache Airflow.
• Knowledge of Azure Data Lake Storage Gen2.
• Experience with Unity Catalog and Databricks Workflows.
• Exposure to Azure DevOps and GitHub Actions.
• Experience working with financial data platforms and enterprise data warehouses.
• Relevant Databricks, dbt, or Microsoft Azure certifications.

Experience Required

10+ Years of Overall Data Engineering Experience

Core Technologies: Azure Databricks, dbt, PySpark, Spark SQL, Advanced SQL, Delta Lake, Azure Data Factory, and Git.

Domain: Capital Markets / Investment Banking / Financial Services.