Data Engineer (Databricks)
Posted 7
views
Required Technical Skills — Databricks
- Unity Catalog — catalogue/schema design, access control, and data lineage
- Lakeflow / Delta Live Tables for pipeline orchestration; Delta Lake table format
- Databricks SQL and cluster/workspace administration (compute policies, pools, cost management)
- Databricks Asset Bundles (DABs) and Databricks Repos for CI/CD
- PySpark / Spark SQL for large-scale data transformation
- Working knowledge of Databricks system tables (audit logs, billing/usage, query history) for observability
- Minimum 5 years of hands-on experience in Data Engineering, Data Platform Engineering, or related disciplines.
- Minimum 3 years of hands-on experience with Databricks involving data pipeline development, platform administration, governance, and optimization.
Required Technical Skills — Common
- Strong SQL and Python (PySpark or general-purpose) for data engineering
- Data modeling — dimensional design, star/snowflake schemas, semantic layers
- CI/CD pipelines (e.g., Azure DevOps, GitHub Actions, GitLab CI) for data engineering workflows
- Infrastructure-as-code (Terraform preferred) for provisioning cloud data platform resources
- Hands-on experience on at least one hyperscaler — AWS, Azure, or Google Cloud
Understanding of data security and compliance frameworks applicable to government/public-sector environments