Intermediate Data Engineer

Key Responsibilities

Data Engineering & Platform Development

• Design, build, and maintain scalable data pipelines and workflows.
• Develop and optimise data processing solutions using Databricks and Apache Spark.
• Build and support end-to-end data integration and transformation processes.
• Ensure data solutions are reliable, efficient, and aligned with best-practice engineering standards.

Cloud & Big Data Solutions

• Develop solutions within Azure cloud environments.
• Support the implementation and optimisation of modern data architectures.
• Work with large-scale datasets and distributed computing frameworks.
• Contribute to the ongoing evolution of the organisation's data platform capabilities.

Data Pipeline Automation & CI/CD

• Develop and maintain CI/CD pipelines to support efficient deployment and release processes.
• Implement automation strategies that improve data delivery, quality, and operational efficiency.
• Support monitoring, testing, and optimisation of data engineering workflows.

Collaboration & Solution Delivery

• Work closely with Data Engineers, Developers, Analysts, and Business Stakeholders to deliver high-quality data solutions.
• Participate in solution design discussions and technical planning.
• Contribute to the continuous improvement of data engineering practices and standards.

Innovation & Continuous Learning

• Stay current with emerging technologies and trends within data engineering and cloud platforms.
• Identify opportunities to improve performance, scalability, and maintainability across data solutions.
• Contribute to knowledge sharing and technical excellence within the team.

Requirements

Essential Skills & Experience

• 3-5 years' experience in a Data Engineering, Backend Development, or similar technical role.
• Previous experience working as a Data Engineer.
• Strong experience with Databricks.
• Proven expertise in Apache Spark and cluster computing environments.
• Advanced Python skills, including mandatory PySpark experience.
• Strong SQL development skills.
• Experience developing and maintaining CI/CD pipelines.
• Experience building scalable and reusable data pipelines.
• Solid software engineering and backend development foundations.
• Exposure to Azure cloud services and infrastructure.

Advantageous Experience

• Microsoft Fabric experience.
• Medallion Architecture experience.
• Experience working with modern data warehousing and analytics environments.
• Exposure to cloud-native data engineering practices.

What Success Looks Like

• High-performing and scalable data pipelines are delivered and maintained.
• Data workflows are optimised for reliability, performance, and cost efficiency.
• CI/CD processes support efficient and consistent deployments.
• Strong collaboration with technical and business teams results in high-quality data solutions.
• Best practices for data engineering, cloud development, and automation are continuously adopted and improved.

Apply now!