Databricks Platform Engineer & Unity Catalog
Posted Updated
Description
Databricks Platform Engineer- Unity Catalog and DevOps
Job Type: Contract (6-12 Months) possible extension
Location: Philadelphia, PA / Vienna, VA - Hybrid / Open to remote job
Job description:
Job summary:
- Program: Consolidation of five legacy data and BI platforms onto Azure Databricks and Power BI
- We are seeking a platform engineer to build and automate an Azure Databricks environment governed by Unity Catalog.
- Working under the platform lead, you will configure catalogs and permissions as code, build deployment pipelines, and support data engineering teams as they migrate workloads onto the platform.
Key responsibilities:
- Configure Unity Catalog objects as code: catalogs, schemas, external locations, storage credentials and grants.
- Implement access controls, including row filters and column masks, to the lead's design.
- Build and maintain CI/CD pipelines for platform configuration, pipelines and jobs.
- Manage repositories, branching and release promotion across environments.
- Automate workspace, cluster policy and job configuration.
- Monitor usage, cost and access using Databricks system tables.
- Troubleshoot access, deployment and integration issues for engineering teams.
- Write runbooks and support testing and production deployments.
Required qualifications:
- 5+ years in data platform, DevOps or data engineering roles, including 2+ years on Databricks.
- 1+ year of hands-on Unity Catalog configuration in a production environment.
- Terraform, or Databricks Asset Bundles, for deploying Databricks resources.
- CI/CD on Azure DevOps or GitHub Actions, with Git branching and pull-request workflows.
- Working Python and SQL.
- Azure fundamentals: Entra ID, ADLS Gen2, Key Vault.
Preferred qualifications:
- Experience in financial services or another regulated industry.
- PySpark, Delta Lake and Lakeflow.
- Databricks Data Engineer Associate certification.
- Experience working in an Agile team.
- AI coding and analysis tools may be used only on endpoints the client has approved.
- Experience in financial services, wealth management or another regulated industry is strongly preferred.
Skills
Sign in to read and join the discussion.