Senior / Lead Data Engineer - DataBricks
Summary
A senior/lead data engineer owns the design, optimisation and governance of scalable Databricks lakehouse pipelines (PySpark, Spark SQL, Delta Lake, Unity Catalog), acting as technical lead who sets standards, mentors engineers, and drives reliability and cost/performance optimisation. Fixed-term contract in Dublin with a view to permanency.
Owns the design and optimisation of scalable data pipelines and lakehouse solutions on Databricks, enabling analytics and data-driven decision-making across the organisation. Acts as a technical leader — setting standards, mentoring engineers, and shaping the platform's evolution.
This is a fixed term contract initially with a view to permanency.
Core Competencies
- Technical leadership and architectural decision-making on Databricks
- Delivery of business value through scalable lakehouse solutions
- Strong stakeholder communication and cross-team collaboration
- Continuous improvement and engineering excellence
Key Responsibilities
- Pipeline Engineering — Design and own scalable ETL/ELT pipelines (PySpark, Spark SQL, Delta Live Tables) following medallion architecture, with strong reliability, monitoring, and cost/performance optimisation.
- Data Modelling & Lakehouse — Govern Delta Lake models and modelling standards across Bronze/Silver/Gold layers, ensuring data quality, integrity, and lifecycle management.
- Platform & Architecture — Own Databricks platform architecture and tooling decisions (Delta Lake, Unity Catalog, Databricks SQL, structured streaming), aligned with enterprise security and governance.
- Reliability & Observability — Define SLAs and implement observability (lineage, monitoring, metrics); resolve performance bottlenecks and manage cost.
- Leadership & Collaboration — Act as technical lead for the data engineering function; mentor engineers; lead design reviews and architecture discussions.
- Governance & Auditability — Ensure traceable, well-governed data processes via Unity Catalog; maintain documentation and support compliance.
- Partner Management — Oversee external delivery partners, ensuring quality, standards adherence, and knowledge transfer.
Experience & Skills
- 5+ years in data engineering, with 3+ years hands-on on Databricks
- Strong SQL and PySpark/Spark SQL expertise
- Proven experience with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL
- Solid grasp of medallion architecture, ETL/ELT design, and Spark performance tuning
- CI/CD experience (Databricks Asset Bundles, Git/Repos)
- Familiarity with structured streaming, Auto Loader, and batch/near-real-time processing
- Databricks certification (Data Engineer Associate/Professional) desirable
- Must have the relevant Visa for a full time position and live within a commutable distance to Dublin
Education
Degree in Computer Science, Data Engineering, or related field