Senior Data Engineer
Summary
Designs and builds scalable data pipelines on Azure and Databricks to integrate and transform enterprise data for analytics and reporting.
The Senior Data Engineer is responsible for designing, building, and managing scalable data pipelines and data platform solutions within the Enterprise Data Platform (EDP) using Azure and Databricks. The role focuses on enabling reliable, governed, and high performance data integration and transformation across multiple source systems to support analytics, reporting, and advanced data use cases.
Duties and Responsibilities
- Design, develop, and maintain scalable data pipelines using Azure Databricks, Azure Data Factory, and related services
- Implement ETL/ELT processes to ingest, transform, and load data into the Enterprise Data Platform
- Develop batch and near real-time data processing solutions
- Apply medallion architecture (bronze, silver, gold layers) for structured data processing
- Ensure data quality, consistency, and integrity across pipelines and datasets Platform Operations & Optimization
- Monitor, maintain, and optimize data pipelines, workflows, and jobs in Databricks and Azure
- Troubleshoot pipeline failures, performance issues, and data inconsistencies
- Optimize Spark jobs, query performance, and storage utilization (e.g., Delta Lake optimization)
- Ensure high availability, scalability, and reliability of data services
- Maintain documentation and ensure compliance with data governance, security, and operational standards
Technical Competencies and Skills
- Data Engineering & Databricks, Azure Data Platform, Data Modeling & Performance Optimization, Data Governance & Security, DevOps & Automation
Education, Trainings and Licenses Required
- Bachelor's degree in Computer Science, Information Technology