Data Engineer

Hands-on Data Engineers to build and optimize data pipelines on the Databricks Lakehouse platform as part of a large-scale enterprise data program. The role is delivery-focused and requires deep practical experience with Apache Spark and the Databricks ecosystem. The successful candidate will develop metadata-driven ingestion and transformation frameworks, implement streaming and incremental processing patterns, and tune pipelines for performance and cost across a broad enterprise source landscape.

Requirements:

• Strong hands-on experience with Apache Spark, including PySpark and Spark SQL.
• Hands-on implementation experience with Databricks Spark Declarative Pipelines (SDP).
• Experience designing and implementing metadata-driven ETL/ELT frameworks using Databricks and SDP.
• Practical experience with structured streaming, materialized views, and streaming/incremental data processing on Databricks.
• Working experience with Delta Lake, including schema evolution, incremental processing, and table optimization.
• Strong understanding of ETL dependency management, orchestration, and pipeline parameterization.
• Experience with Databricks performance optimization: Spark tuning, partitioning, and efficient data processing.
• Implementation knowledge of workspace configuration, compute/cluster configuration, Unity Catalog configuration, permissions, and policies.