Data Engineer | AI & Analytics
Summary
Data Engineer at a management consulting firm building scalable production data pipelines from AI prototypes using Azure Data Factory and Python, integrating ERP, databases, Excel, and flat files across client engagements.
Role Summary
Requirements
- Own pipeline implementation end-to-end: environment and connectivity setup, build, validation, deployment and go-live support
- Translate business rules and AI-generated prototypes from consulting teams into validated, production-ready pipelines
- Integrate diverse sources: ERP systems, databases, external data, non-standard Excel and flat files
- Define and automate tests including schema checks, row-count and value reconciliations, and failure alerts
- Review SQL data models and database design; recommend improvements for scalability, performance and maintainability
- Manage development and production deployments; monitor and troubleshoot pipelines in production
- Work directly with client IT teams on connectivity, access, security and deployment requirements
- 4–7 years of relevant experience in data engineering
- Strong hands-on experience with Azure Data Factory (ADF) (mandatory)Hands-on experience with Azure Self-Hosted Integration Runtime (SHIR) (mandatory)
- Strong Python skills, including data validation, reconciliation and source-to-target parity matching
- Strong grounding in SQL data modelling and database design fundamentals
- Experience with Azure SQL and Microsoft Entra ID (preferred)