Senior Data Engineer - Databricks
Posted Updated
Description
We are looking for a Senior Data Engineer with strong hands-on experience in Databricks and a solid understanding of Data Governance concepts. The ideal candidate should have 7+ years of overall experience, including a minimum of 5 years of Databricks experience, covering data engineering and/or Databricks infrastructure, along with knowledge of data governance principles and practices.
Job Responsibilities:
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
- Work on Databricks infrastructure, including workspace configuration, clusters, compute resources, access controls, and performance optimization.
- Apply Data Governance principles, including data quality, data lineage, metadata management, data cataloging, and access controls.
- Work with Unity Catalog to manage data permissions, governance policies, and data assets.
- Develop and manage Delta Lake tables using Medallion Architecture (Bronze, Silver, and Gold).
- Integrate data from multiple sources, including databases, Amazon S3, files, and REST APIs.
- Create, schedule, monitor, and optimize Databricks Jobs and Workflows.
- Implement data validation, error handling, logging, monitoring, and data quality checks.
- Optimize Spark workloads, SQL queries, and data processing pipelines for performance and scalability.
- Support data discovery, metadata documentation, and governance standards across data platforms.
- Collaborate with cross-functional teams to ensure data accuracy, consistency, security, and compliance.
- Troubleshoot data pipeline and infrastructure issues and recommend improvements to platform reliability.
- Contribute to enterprise data management and governance best practices.
Basic Qualification:
- 7+ years of overall experience in Data Engineering or a related data technology role.
- Minimum 5 years of hands-on Databricks experience, including data engineering and/or Databricks infrastructure.
- Mandatory understanding of Data Governance concepts, including data quality, metadata management, data lineage, data cataloging, data ownership, and access controls.
- Practical knowledge of data governance principles and their application in data platforms.
- Understanding of data governance frameworks, policies, standards, and data management best practices.
- Experience or knowledge of implementing data quality checks, data lineage tracking, metadata documentation, and governed data access.
- Strong expertise in Python, PySpark, and Advanced SQL.
- Hands-on experience with the Databricks Lakehouse Platform, Delta Lake, Unity Catalog, Workflows, Jobs, Clusters, and Notebooks.
- Experience developing ETL/ELT pipelines, data transformations, batch processing, and incremental data loading.
- Good understanding of data warehousing and data modeling concepts.
- Knowledge of data security, access management, and governance practices within data platforms.
- Experience with Spark performance tuning, partitioning, file optimization, and query optimization.
- Familiarity with Git, CI/CD practices, and production deployment processes.
- Strong analytical, problem-solving, and communication skills.
Skills
Sign in to read and join the discussion.