Data Engineer (Databricks)
Posted Updated 4
views
Summary
Senior data engineer to build and administer a Databricks data platform, likely for a government/public-sector client: day to day covers Unity Catalog governance, Delta Lake/Live Tables pipelines, PySpark transformations, CI/CD via Databricks Asset Bundles, and IaC (Terraform) on AWS/Azure/GCP. Requires 5+ years in data engineering and 3+ with Databricks.
Required Technical Skills - Databricks
Unity Catalog - catalogue/schema design, access control, and data lineage
Lakeflow / Delta Live Tables for pipeline orchestration Delta Lake table format
Databricks SQL and cluster/workspace administration (compute policies, pools, cost management)
Databricks Asset Bundles (DABs) and Databricks Repos for CI/CD
PySpark / Spark SQL for large-scale data transformation
Working knowledge of Databricks system tables (audit logs, billing/usage, query history) for observability
Minimum 5 years of hands-on experience in Data Engineering, Data Platform Engineering, or related disciplines.
Minimum 3 years of hands-on experience with Databricks involving data pipeline development, platform administration, governance, and optimization.
Required Technical Skills - Common
Strong SQL and Python (PySpark or general-purpose) for data engineering
Data modeling - dimensional design, star/snowflake schemas, semantic layers
CI/CD pipelines (e.g., Azure DevOps, GitHub Actions, GitLab CI) for data engineering workflows
Infrastructure-as-code (Terraform preferred) for provisioning cloud data platform resources
Hands-on experience on at least one hyperscaler - AWS, Azure, or Google Cloud
Understanding of data security and compliance frameworks applicable to government/public-sector environments