Data Engineering Consultant
Summary
A consultant role in Doha designing, building, and supporting batch, streaming, and API-based data pipelines and integrations for client data and analytics solutions. Day to day involves SQL/programming, cloud data platforms, data quality controls, documentation, and handing over maintainable pipelines to client teams.
Role Purpose
The Data Engineering Consultant designs, builds and supports reliable data pipelines, integrations and services for client data and analytics solutions. The role converts approved architecture, business requirements and governance controls into tested technical components that move, transform and deliver data with appropriate quality, metadata, lineage, security and operational monitoring. The consultant works with architects, analysts, platform teams, security specialists and client system owners throughout design, development, testing and transition. The role may implement within client environments under authorised access, but production ownership and operational authority remain with the client. The expected outcome is maintainable engineering work with clear source-to-target logic, automated controls, deployment evidence, runbooks and knowledge transfer suitable for client operation after handover.
Key Responsibilities
Analyse approved requirements, source systems, target models, data volumes, schedules, controls and service expectations. Design and build batch, streaming or API-based data ingestion and transformation pipelines. Implement source-to-target mappings, business transformations, reference-data logic and data-standardisation rules. Embed validation, reconciliation, exception handling and data-quality controls within engineering workflows. Capture technical metadata, lineage, schedules, dependencies, ownership and operational information for delivered pipelines. Apply approved security, privacy, classification, access, encryption and logging requirements. Develop automated unit, integration, regression and data-validation tests and retain execution evidence. Optimise performance, scalability, reliability and cost within the approved architecture and platform constraints. Implement deployment automation, version control, configuration management and controlled release practices. Monitor pipeline operation, investigate failures and support defect resolution during implementation and transition. Prepare technical specifications, code documentation, support procedures, runbooks and handover records. Walk client engineers through design decisions, operating procedures and known limitations before transition.
Minimum Requirements
Education
Bachelor's degree in computer science, software engineering, data engineering or a related field. An equivalent combination of relevant education and directly applicable consulting or implementation experience may be considered.
Professional Experience
Typically 3-8 years in data engineering, integration, ETL/ELT, software or cloud data platforms. Experience in data engineering, integration, ETL/ELT, software development or cloud data delivery. Experience building and testing production-grade pipelines, APIs or data-processing services. Experience with version control, automated deployment, monitoring and operational support. Experience working from architecture, security and data-quality requirements in multidisciplinary teams.
Technical and Domain Knowledge
Data ingestion, transformation and orchestration patterns. SQL and one or more relevant programming or scripting languages. Batch, streaming, API and integration technologies. Cloud or enterprise data platforms and storage patterns. Data quality, reconciliation, exception handling and test automation. Metadata, lineage, source-to-target mapping and documentation. Security, access control, encryption, logging and secrets management. Version control, CI/CD, configuration and operational monitoring.
Core Competencies
SQL and programming. Data pipelines. APIs and integration. Cloud/data platforms. Testing and observability. Security and quality controls. Engineering discipline. Problem solving. Reliability focus. Automation mindset. Clarifying technical requirements and identifying missing decisions early. Estimating engineering effort, dependencies and delivery risk. Explaining technical designs and defects to mixed audiences. Producing maintainable documentation and auditable test evidence. Collaborating with client engineers, vendors and multidisciplinary teams. Working within controlled client access and change processes. Transferring engineering knowledge and supporting operational handover.
Preferred Certifications
Google Cloud Professional Data Engineer or comparable cloud data credential CDMP Associate or Practitioner
Focus areas
SQL and programming Data pipelines APIs and integration