Databricks Solution Architect
Description
Consultancy client are looking to recruit a Databricks Solution Architect to define and deliver enterprise-scale data platform architectures using Azure Databricks.
The candidate must possess deep hands-on Databricks engineering knowledge alongside experience designing secure, scalable and highly available data platforms for UK Public Sector organisations. This is a technical architecture role requiring the ability to move between architecture design, technical implementation and engineering leadership.
Essential Technical Skills
- 5+ years of hands-on Databricks platform experience.
- Extensive Azure Databricks architecture experience.
- Expert-level PySpark, Spark SQL and Python knowledge.
- Deep understanding of Apache Spark processing architecture.
- Extensive Delta Lake and Lakehouse architecture experience.
- Strong experience designing Medallion Architecture.
- Expert knowledge of Unity Catalog, governance, permissions and data lineage.
- Experience designing Databricks workspaces, clusters, compute policies and networking.
- Knowledge of private endpoints, managed identities, Key Vault and secure connectivity.
- Experience with Azure Data Lake Storage Gen2, Azure Data Factory and Azure services.
Public Sector Experience
Extensive experience delivering enterprise data architectures
Understanding of UK Government security principles and data governance requirements.
Experience architecting platforms handling sensitive and restricted government data.
Knowledge of UK GDPR, Data Protection Act 2018 and relevant government security standards.
Experience producing architecture documentation, HLDs, LLDs and technical design authorities.
Experience working with public sector enterprise architecture and governance boards.
Security Clearance Requirements
Be eligible and willing to undergo SC clearance
Previous experience working on sensitive government projects is advantageous.
Responsibilities
- Own the end-to-end Databricks solution architecture.
- Define scalable and secure Lakehouse architecture.
- Establish engineering and architecture standards.
- Design data ingestion, transformation and consumption patterns.
- Define Unity Catalog and enterprise data governance models.
- Provide technical direction to data engineering teams.
Skills
Sign in to read and join the discussion.