Senior / Lead Data Engineer - DataBricks

Owns the design and optimisation of scalable data pipelines and lakehouse solutions on Databricks, enabling analytics and data-driven decision-making across the organisation. Acts as a technical leader — setting standards, mentoring engineers, and shaping the platform's evolution. This is a fixed term contract initially with a view to permanency.

Core Competencies

• Technical leadership and architectural decision-making on Databricks
• Delivery of business value through scalable lakehouse solutions
• Strong stakeholder communication and cross-team collaboration
• Continuous improvement and engineering excellence

Key Responsibilities

• Pipeline Engineering — Design and own scalable ETL/ELT pipelines (PySpark, Spark SQL, Delta Live Tables) following medallion architecture, with strong reliability, monitoring, and cost/performance optimisation.
• Data Modelling & Lakehouse — Govern Delta Lake models and modelling standards across Bronze/Silver/Gold layers, ensuring data quality, integrity, and lifecycle management.
• Platform & Architecture — Own Databricks platform architecture and tooling decisions (Delta Lake, Unity Catalog, Databricks SQL, structured streaming), aligned with enterprise security and governance.
• Reliability & Observability — Define SLAs and implement observability (lineage, monitoring, metrics); resolve performance bottlenecks and manage cost.
• Leadership & Collaboration — Act as technical lead for the data engineering function; mentor engineers; lead design reviews and architecture discussions.
• Governance & Auditability — Ensure traceable, well-governed data processes via Unity Catalog; maintain documentation and support compliance.
• Partner Management — Oversee external delivery partners, ensuring quality, standards adherence, and knowledge transfer.

Experience & Skills

• 5+ years in data engineering, with 3+ years hands-on on Databricks
• Strong SQL and PySpark/Spark SQL expertise
• Proven experience with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL
• Solid grasp of medallion architecture, ETL/ELT design, and Spark performance tuning
• CI/CD experience (Databricks Asset Bundles, Git/Repos)
• Familiarity with structured streaming, Auto Loader, and batch/near-real-time processing
• Databricks certification (Data Engineer Associate/Professional) desirable
• Must be based within a commutable distance to Dublin
• Must have the relevant work status to work in Ireland.

Education

Degree in Computer Science, Data Engineering, or related field