Senior Databricks & AI Architect – Sydney
Summary
Leads Databricks-based data platforms and Agentic AI solutions for insurance processes like claims and underwriting, optimizing Spark architectures, FinOps, and regulatory-compliant data governance.
Delivery Centric is seeking a Senior Databricks & AI Architect to lead modern data platform and AI initiatives. The role focuses on scalable Spark architectures, FinOps and optimisation, and Agentic AI solutions for high-value insurance processes such as claims, underwriting, and risk assessment.
Key Responsibilities
- Architectural Leadership: Design and govern end-to-end data architectures on the Databricks Data Intelligence Platform, ensuring seamless integration with existing insurance ecosystems.
- AI Innovation: Lead the design of emerging AI workflows, specifically focusing on Agentic AI (multi-agent systems), LLM orchestration (LangChain/LlamaIndex), and RAG patterns tailored for insurance policy and regulatory documents.
- Databricks FinOps & Performance: Own the platform’s efficiency. Implement cluster policies, Photon engine optimisations, and cost-attribution models to ensure high performance with a focus on ROI.
- Data Engineering Excellence: Standardise data engineering templates (Medallion architecture) and implement robust data modelling (Star/Snowflake, Vault 2.0) to handle structured and unstructured insurance data.
- Domain Alignment: Translate complex insurance concepts (Premiums, Claims, Actuarial models) into scalable technical schemas and AI logic.
- Governance & Security: Ensure all designs comply with industry regulations using Databricks Unity Catalog.
Required Skills & Experience
- Experience: 8+ years in Data Engineering/Architecture, with at least 3+ years of hands-on leadership within the Databricks ecosystem.
- Core Tech: Expert-level knowledge of the Spark framework (PySpark/Scala), Delta Lake, and Databricks SQL.
- AI/ML Expertise: Proven experience building and deploying AI solutions. Deep understanding of the Agentic AI landscape—moving beyond simple chatbots to autonomous, goal-oriented agents.
- FinOps Mastery: Specific experience in Databricks cost management, instance selection, and performance tuning (Z-Ordering, Liquid Clustering).
- Data Management: Strong background in data modelling and the creation of reusable ETL/ELT templates and frameworks.
- Insurance Domain: Prior experience in the Insurance or FinServ sector is highly preferred; you must understand the nuances of policy-centric data.
Ready to take on a Senior Databricks & AI Architect role in Sydney with Delivery Centric? Apply now to join a team that delivers excellence and realises value for its clients.