Data Engineer - Databricks & Automation
Description
Key Responsibilities
Design, develop and maintain scalable ETL/ELT data pipelines using Databricks, Python, PySpark, SQL and Delta Lake.
Integrate data from multiple sources, including databases, APIs and files, with streaming ingestion where required.
Develop and maintain curated datasets and data models to support business reporting, analytics and AI applications.
Implement data quality checks, data validation, reconciliation and incremental data processing.
Monitor data pipelines, investigate failures, troubleshoot issues and ensure data accuracy and availability.
Optimise data processing performance, pipeline execution and compute utilisation.
Implement version control, automated testing, CI/CD and technical documentation.
Identify and automate repetitive data engineering and support activities.
Explore practical AI
Skills
Sign in to read and join the discussion.