Data Engineering Advisor

Job Description

Job Description

Job Title: Data Engineering Advisor

Location: Montreal, QC

Duration: 12 Months

The data engineering consultant contributes to the development of solutions for collecting, integrating, consolidating, cleaning, storing, and processing data streams. They participate in the development, maintenance, support, and operation of IT applications and data pipelines, performing functional analysis, technical design, coding, debugging, and program documentation.

Responsibilities:

• Gather the analytical needs expressed by users

• Designing and building data collection systems and platforms for a variety of projects

• Develop data pipelines, integrate, transform, and consolidate data from different systems into context-appropriate structures.

• Perform data mapping specification and documentation

• Implement, monitor, maintain and optimize diverse systems that meet business needs

• Use various scripting languages and tools to enable the storage and production of complex datasets

• Developing algorithms, prototypes, and proof-of-concept demonstrations within complex projects

• Identify opportunities to optimize systems based on trends and behaviors and recommend improvements to data management services

Requirements and qualifications

• Azure Cloud & Services: AI/ML Services, Azure Functions, Azure Data Lake, Azure API Management

• NLP & Generative AI: OpenAI, fine-tuning, prompt engineering, model integration

• DevOps & CI/CD: Azure DevOps/GitHub Actions

• Infrastructure as Code (IaC): Terraform, Provider Azure RM, Terraform Test Framework

• MLOps & LLMOps Deployment: MLOps, LLMOps (integration and industrialization of models)

• ETL/ELT & data integration: Azure Data Factory, Medallion architecture (Bronze/Silver/Gold)

• Monitoring & Observability: Azure Monitor, Azure Log Analytics, Splunk, Dynatrace

• Security & Access: Azure KeyVault (secrets management), Entra ID, Azure AD,

• Resilience & Disaster Recovery: RTO, RPO, replication, failover and fallback concepts

• API: Design and deployment of REST APIs

• Programming languages: Python, R

• Scripting & Automation: PowerShell, Bash

• Having a good knowledge of the sector

• Demonstrate strategic thinking

• Mastering interpersonal relationships