Sr. Data Engineer
Summary
Designs and maintains scalable data pipelines using Azure Databricks and Azure Data Factory to ingest, transform, and optimize data for analytics and reporting, ensuring data quality, governance, and high performance.
Job Summary
The Senior Data Engineer is responsible for designing, building, and managing scalable data pipelines and data platform solutions within the Enterprise Data Platform (EDP) using Azure and Databricks. The role focuses on enabling reliable, governed, and high-performance data integration and transformation across multiple source systems to support analytics, reporting, and advanced data use cases.
Duties and Responsibilities
This position combines hands-on data engineering, platform optimization, and stakeholder coordination, ensuring the delivery of robust, secure, and business-aligned data solutions while supporting data governance and best practices.
Data Engineering & Pipeline Development
Design, develop, and maintain scalable data pipelines using Azure Databricks, Azure Data Factory, and related services
Implement ETL/ELT processes to ingest, transform, and load data into the Enterprise Data Platform
Develop batch and near real-time data processing solutions
Apply medallion architecture (bronze, silver, gold layers) for structured data processing
Ensure data quality, consistency, and integrity across pipelines and datasets
Platform Operations & Optimization
Monitor, maintain, and optimize data pipelines, workflows, and jobs in Databricks and Azure
Troubleshoot pipeline failures, performance issues, and data inconsistencies
Optimize Spark jobs, query performance, and storage utilization (e.g., Delta Lake optimization)
Ensure high availability, scalability, and reliability of data services
Maintain documentation and ensure compliance with data governance, security, and operational standards
Team Management
Provide guidance to data engineers and developers on best practices in Azure and Databricks
Review code, pipelines, and technical designs to ensure quality and standards compliance
Support the team in resolving technical issues and improving delivery efficiency
Promote reusable frameworks, templates, and automation practices
Mentor team members on data engineering concepts and cloud-based architectures
Stakeholder Engagement
Collaborate with business units, analytics teams, and platform engineers to gather and validate data requirements
Translate business needs into scalable data models and pipelines
Act as liaison between source system owners, data consumers, and platform teams
Communicate pipeline status, issues, and improvements to stakeholders
Support analytics and BI teams by ensuring availability of clean and reliable datasets
Technical Competencies and Skills
Data Engineering & Databricks, Azure Data Platform, Data Modeling & Performance Optimization, Data Governance & Security, DevOps & Automation
Education, Trainings and Licenses Required
Bachelor's degree in Computer Science, Information Technology