Senior Data Engineer / Databricks Developer
Summary
Build and maintain Databricks-native data quality monitoring, pipelines, and dashboards using Spark, Delta Lake, and Python to ensure reliable reporting and governance.
Responsibilities
- Design and implementation of a Databricks-native Data Quality Monitoring framework
- Configuration and implementation of DQX-based data quality rules
- Development of data pipelines and data models supporting monitoring and reporting
- Creation of dashboards and trending views for business users and data owners
- Linking data quality rules, data usage, and critical data points
- Documentation of technical design, rule logic, and operating model
- Supporting the scaling of the solution to additional data domains over time
- Working closely with data owners, business users, and technical stakeholders
Requirements
- Strong hands‑on experience with Databricks
- Experience with Spark, SQL, Delta Lake, and Python
- Experience designing and implementing data pipelines / ETL-ELT
- Experience with data quality frameworks, preferably DQX
- Understanding of data governance, data ownership, and rule‑based data quality monitoring
- Ability to translate business requirements into scalable technical solutions
- Ability to work independently and take end‑to‑end ownership