Senior Data Engineer - AI-Driven Data Pipelines
Posted Updated
Description
Management Solutions International is seeking a Senior Data Engineer to own the data stack for a Doha client engagement and extend the data model with new sources. You will ensure data reliability for AI workflows and lead data integration efforts with Databricks and Azure Data Factory on an Azure-hosted platform.
You will join the founding team in Doha, collaborate with the Senior Solutions Architect, and contribute to AI-driven financial technology solutions while upholding agile engineering
Design, develop, and implement robust, efficient, and scalable data pipelines into the Client's data platform. Ingest structured data feeds, APIs, and unstructured documents. Build batch pipelines for pre-populated datasets. Build on-demand enrichment pipelines triggered by user requests and business requirements. Build and operate pipelines using Databricks and Azure Data Factory. Ensure all pipelines include unit testing, automation, and CI/CD. Develop production-grade pipelines that are tested, automated, repeatable, and scalable from the outset. Master's degree in Computer Science, Engineering, or a related field. 7+ years of significant Data Engineering experience, excluding internships. Excellent verbal and written English communication skills. Outstanding programming skills in PySpark, SQL, and Python. Experience processing data at terabyte scale in production environments. Hands-on experience with Azure Cloud services, particularly Databricks, Azure Data Factory, and Azure DevOps. Familiar with document databases such as Cosmos DB and MongoDB, graph databases such as Neo4j, and ELK technologies. Strong understanding of SQL and NoSQL database internals, query languages, and distributed computing fundamentals. Experience building or operating entity-resolution and record-linkage solutions across messy and complex data sources. Experience with fuzzy matching, blocking, and measuring match quality through precision and recall. Has designed schemas or ontologies for knowledge graphs or linked-data environments, including entity-relationship modelling. Has delivered data integration for external clients or partners, including data contracts, secure data transfer, and access controls. Learns quickly and demonstrates a continuous-improvement mindset. Is highly collaborative and team-oriented. Is proactive in identifying data issues and trade-offs and presenting recommendations. Is comfortable working directly with Client teams and explaining complex data issues to non-technical stakeholders. Can work autonomously within a small team while maintaining close collaboration with wider technical teams. Experience working in regulated or sovereign environments with data-residency requirements would be advantageous. Experience with financial services, private markets, energy, or sustainability data would be valuable; however, a proven track record of delivering reliable, production-grade data engineering solutions is more important than experience in any specific domain.Skills
Sign in to read and join the discussion.