Cloud Data Engineer
Summary
Designs and builds scalable Azure-based data pipelines and cloud-native solutions for a bank, using ADF, Databricks, Synapse, and APIs to enable analytics and AI.
Role Purpose
The Cloud Data Engineer is responsible for designing, developing and maintaining enterprise-grade cloud data solutions that enable advanced analytics, reporting and business intelligence across the bank.
The role focuses on delivering scalable, secure and high-quality data products on Microsoft Azure by building robust ETL/ELT pipelines, integrating multiple data sources through APIs, and ensuring reliable data availability for downstream consumers. The successful candidate will provide technical leadership in cloud data engineering while promoting engineering best practices, automation and modern data architecture.
Key Responsibilities
Cloud Data Engineering
- Design, develop and maintain scalable cloud-native data solutions on Microsoft Azure.
- Build enterprise-grade data products for analytics, reporting and AI initiatives.
- Develop reliable ETL/ELT data pipelines to ingest, transform and publish data.
- Optimise data processing for performance, scalability and cost efficiency.
- Implement reusable engineering patterns and data engineering standards.
Azure Data Platform
- Develop solutions using Azure Data Factory (ADF).
- Build data processing pipelines using Azure Databricks.
- Develop solutions on Azure Synapse Analytics.
- Work with Azure Data Lake Storage (ADLS Gen2).
- Integrate Azure SQL Database, SQL Server and other enterprise data sources.
- Support cloud migration and modernisation initiatives.
Data Integration
- Design and implement API integrations.
- Consume REST APIs and enterprise services.
- Integrate structured and semi-structured data sources.
- Develop secure and scalable ingestion frameworks.
- Automate data ingestion from multiple internal and external platforms.
Data Quality & Governance
- Ensure data accuracy, consistency and completeness.
- Implement monitoring and alerting for pipeline failures.
- Perform root cause analysis of data quality issues.
- Support metadata management and data governance initiatives.
- Ensure compliance with security and regulatory requirements.
Advanced Analytics Enablement
- Prepare clean, trusted datasets for Data Scientists and BI teams.
- Optimise datasets for reporting and advanced analytics.
- Support machine learning and AI workloads.
- Enable self-service analytics capabilities.
Cloud Engineering Best Practices
- Participate in solution design and architecture discussions.
- Implement CI/CD pipelines for data engineering deployments.
- Use Infrastructure as Code where appropriate.
- Follow DevOps and Agile delivery methodologies.
- Perform code reviews and mentor junior engineers.