Data & AI Engineer

Must Have Technical / Functional Skills

• Strong core data engineering competencies with expertise in data concepts and data modelling
• Experience with Big Data platforms and scalable data pipeline design
• Hands-on experience with Microsoft Fabric Data Agents and Azure AI Services
• Knowledge of AI/ML integration within enterprise data platforms
• Strong analytical and problem-solving capabilities
• Expertise in performance tuning, monitoring, and optimization
• Proficiency in PySpark for large-scale data processing
• Ecommerce domain knowledge with customer behavior analytics experience
• Experience with Adobe Analytics, Adobe Customer Journey Analytics (CJA), and clickstream data analysis

Roles & Responsibilities

Experimentation Data Enablement (Silver Layer Ownership)

• Design, build, and maintain curated Silver-layer datasets in Microsoft Fabric for experimentation reporting and analytics
• Collaborate with BI/Data Reporting teams to define dimensions, metrics, and joins such as visitor/session, variant, campaign, geo, device, channel, funnel steps, and conversion events
• Develop reusable and standardized data products including tables and views for dashboards, scorecards, and ad hoc reporting
• Ensure Silver-layer datasets are clean, conformed, deduplicated, and aligned with agreed business definitions

Data Gap Analysis & Assessment

• Conduct regular gap assessments across experimentation requirements, existing Silver-layer data, and upstream telemetry/source systems
• Identify missing fields, inconsistent definitions, data latency issues, and join-key mismatches
• Document business impact, severity, remediation plans, timelines, and dependencies for identified issues
• Recommend improvements in data models, including facts/dimensions, surrogate keys, grain definition, and conformance rules

Gold Layer Requirements & Stakeholder Management

• Lead workshops with experimentation, BI, measurement, and engineering teams to define Gold-layer reporting requirements
• Define KPI calculations, attribution rules, scorecard structures, segmentation requirements, governance standards, and refresh SLAs
• Prepare functional and technical documentation including source-to-target mappings, data dictionaries, validation rules, and acceptance criteria
• Ensure alignment on single-source-of-truth definitions across CJA, Power BI, and scorecards

Data Pipeline Engineering

• Build and maintain robust pipelines using Microsoft Fabric Pipelines and Azure Data Factory (ADF)
• Work with 1DS telemetry pipelines or equivalent systems to ensure accurate event and attribute flow into Fabric
• Implement orchestration, incremental loads, monitoring, and error-handling mechanisms to meet reporting timelines

Data Validation & Reconciliation

• Perform reconciliation between Silver/Gold datasets and Customer Journey Analytics (CJA)
• Validate event counts, session/user logic, experiment attribution, conversions, and time-window consistency
• Build automated checks for missing data, duplicate events, schema drift, and metric anomalies
• Coordinate issue resolution with telemetry, tagging, product engineering, and reporting teams

Experimentation Lifecycle Support

• Ensure datasets are ready for pre-launch checks, measurement, scorecard generation, health checks, and post-test analysis
• Curate experiment metadata including test IDs, allocation details, start/end dates, KPI metrics, and slicing dimensions
• Support consistent and reliable experimentation scorecard generation

AI Agent Design & Development

• Design and develop AI-powered agents using Fabric Data Agents, Copilot, and Azure OpenAI
• Enable automation for scorecard creation, narrative summaries, self-service analytics, anomaly investigation, and metric definition assistance
• Define agent scope, personas, grounding datasets, RBAC/security models, and evaluation metrics
• Partner with experimentation and reporting teams for pilot implementation, feedback gathering, and production rollout

Documentation, Governance & Operational Excellence

• Maintain detailed documentation for datasets, transformation logic, metric definitions, pipelines, validation rules, and operational runbooks
• Establish standards for naming conventions, semantic consistency, versioning, backward compatibility, and performance optimization
• Provide operational support including monitoring, troubleshooting, incident management, and continuous improvement initiatives