Data Engineer (Azure, Databrick)

Data Management & Transformation

• Design, develop and maintain data pipelines that extract data from business applications, databases and external systems.
• Implement data quality checks and validation processes to ensure data accuracy, consistency and completeness.
• Perform data cleansing, aggregation, filtering, transformation and enrichment.
• Establish and maintain governance frameworks for data and algorithms used in analytics, automated decision-making and analytical applications.
• Develop and maintain logical and physical data models, including data structures, relationships and constraints.
• Ensure data solutions comply with security, governance and organisational best practices.

Data Platform Optimisation & Automation

• Develop and maintain CI/CD pipelines for data deployments and cloud resource provisioning.
• Optimise data pipelines and processing workflows for performance, scalability and efficiency.
• Optimise data models and algorithms for data quality, security, governance, performance and scalability.
• Monitor data warehouse and ETL platform capacity, including capacity planning and forecasting.
• Monitor and tune data platforms, troubleshoot performance bottlenecks and implement optimisation techniques such as caching and indexing.
• Monitor platform resource and credit consumption and perform regular housekeeping activities.
• Support the deployment, integration and maintenance of AI/ML solutions within the data platform.

Collaboration & Business Enablement

• Partner with the Data Lead and business users to manage and maximise the value of data as a business asset.
• Work with stakeholders to understand data requirements and translate them into scalable technical solutions.
• Guide business users in the creation and maintenance of reports and dashboards.
• Collaborate with technology, data and business teams to improve data quality, accessibility and usability.
• Any other ad-hoc duties as assigned by supervisor

• 5+ years of experience in data management or data engineering, covering areas such as data integration, data modelling, data quality and platform optimisation.
• Hands-on experience designing and implementing data architectures using Azure and Databricks.
• Strong programming skills in Python, PySpark and SQL.
• Strong experience with relational and non-relational databases, including SQL and NoSQL databases.
• Hands-on experience with Azure DevOps and Databricks LakeFlow Jobs, including version control, CI/CD and pipeline deployment.
• Experience with data catalogue and governance tools such as Databricks Unity Catalog and Microsoft Purview.
• Experience supporting AI/ML model deployment, feature engineering and production inference is an advantage.