Data Engineer

Responsibilities Design and develop scalable backend services that integrate with external vendor systems through RESTful APIs Implement reliable data ingestion services, such as time-window retrieval and pagination Manage API error handling, retry mechanisms, and idempotent processing to ensure dependable data ingestion Build and maintain data pipelines to process high volumes of transactional and event data Carry out data transformation, validation, and enrichment for downstream systems Support both batch and near real-time data processing workflows Develop and deploy applications using Azure cloud services, such as Azure Functions, App Services, Storage, Key Vault, APIM, VNET, etc. Ensure solutions are built to be scalable, resilient, and highly available Implement observability practices, including logging, monitoring, and alerting Ensure data accuracy, consistency, and reconciliation across multiple data sources Support auditability and traceability of data flows to meet regulatory requirements Work with external vendors to define and refine API contracts and integration designs Deliver high-quality, maintainable, and reusable software solutions Contribute to team engineering standards and best practices Support continuous delivery through CI/CD pipelines and automated testing

Qualifications Strong proficiency in Python, preferred, or other backend languages such as Java or Node.js, along with strong proficiency in SQL, such as T-SQL, MySQL, or pgSQL Solid experience in developing backend services and API integrations Experience working with RESTful APIs, JSON payloads, and API consumption patterns Knowledge of secure integration patterns, such as API keys, TLS, certificates, and PGP Strong understanding of: Pagination, filtering, and incremental data retrieval Idempotency, retries, and failure handling in distributed systems

Required Skills Experience building data pipelines, either batch or streaming Familiarity with data transformation, validation, and data modelling Experience handling large datasets and high-volume data ingestion systems Understanding of data reconciliation and audit requirements will be an advantage Experience using data visualization tools such as Databricks Dashboard, Power BI, or Tableau

Preferred Skills Experience with relational databases such as PostgreSQL or MS SQL Knowledge of query optimisation and managing large datasets Experience with cloud platforms, preferably Microsoft Azure Experience setting up CI/CD pipelines and using GitLab and other relevant tools, such as GitLab CI/CD, SHIP, or HATS Familiarity with containerisation, such as Docker, will be an advantage Experience implementing: Logging and monitoring Metrics and alerting Understanding of system reliability and fault-tolerant design Experience integrating with third-party or vendor systems Experience working with event-driven systems or messaging platforms, such as Kafka or Event Hub Exposure to IoT, device data, or telemetry systems