Principal data engineer (databricks)
Summary
Designs and builds scalable ETL pipelines on Databricks, using PySpark, Delta Lake, and Medallion Architecture to move and transform data for production analytics.
Required Certifications Databricks Certified Data Engineer Professional
Role requirements 7+ years of experience building highly reliable ETL pipelines and managing complex data workflows. Elite-level proficiency with Apache Spark, Delta Lake architecture, and deploying optimized data infrastructure in production environments.
Technical requirements
Core Architecture: Delta Lake, Medallion Architecture (Bronze/Silver/Gold), Unity Catalog
Engines: Py Spark, Spark SQL, Photon Engine, Delta Live Tables (DLT)
Orchestration: Databricks Workflows, dbt (Data Build Tool), Airflow
Performance: Z-Ordering, Liquid Clustering, Data Skipping, and Spark UI Debugging
Focus areas Databricks Spark Delta Lake
Role requirements 7+ years of experience building highly reliable ETL pipelines and managing complex data workflows. Elite-level proficiency with Apache Spark, Delta Lake architecture, and deploying optimized data infrastructure in production environments.
Technical requirements
Core Architecture: Delta Lake, Medallion Architecture (Bronze/Silver/Gold), Unity Catalog
Engines: Py Spark, Spark SQL, Photon Engine, Delta Live Tables (DLT)
Orchestration: Databricks Workflows, dbt (Data Build Tool), Airflow
Performance: Z-Ordering, Liquid Clustering, Data Skipping, and Spark UI Debugging
Focus areas Databricks Spark Delta Lake