ETL Test Automation Engineer – Azure Data Platform
Summary
Designs and runs automated tests for Azure-based ETL pipelines, data migrations, and cloud data platforms, ensuring data quality and integrity across enterprise systems.
The QA & Automation Engineer will be responsible for validating ETL pipelines, data migration processes, and cloud data platform integrations across Azure services. This role involves hands‑on testing of large-scale data platforms, automation development, advanced SQL validation, and ensuring end‑to‑end data quality across enterprise data engineering initiatives.
Key Responsibilities
- Perform end‑to‑end testing of ETL pipelines built using Azure Data Factory, Azure Databricks, Azure Synapse, and SSIS.
- Validate data transformations, mappings, data quality rules, and data lineage.
- Conduct source‑to‑target (S2T) reconciliation, data profiling, completeness, and accuracy checks.
- Verify schema changes, incremental loads, delta loads, and historical load processes.
Data Migration Testing
- Design and execute test strategies for large‑scale on‑prem to cloud or cross‑cloud data migration projects.
- Validate ETL/ELT processes and post‑migration data accuracy.
- Perform count checks, checksum validation, CDC validation, duplicate checks, and referential integrity validation.
- Review and validate data mapping documents, business rules, and acceptance criteria.
Data Lake & Cloud Platform Testing
- Test ingestion pipelines into Azure Data Lake Storage (ADLS Gen2) from multiple source systems.
- Validate partitioning, folder structures, file formats (Parquet, CSV, JSON), and governance standards.
- Conduct performance and scalability testing for large data workloads.
- Validate integration flows across Data Lake, Synapse, ADF, and Power BI.
Automation Testing
- Develop automated data validation and regression frameworks using Python, PySpark, and ADF automated validation frameworks.
- Integrate automated tests within CI/CD pipelines using Azure DevOps.
- Build reusable automation templates, accelerators, and validation utilities.
- Create and maintain test plans, scenarios, and test cases in Azure DevOps or Jira.
- Log, track, triage, and validate defects in collaboration with development and data engineering teams.
- Publish daily/weekly QA health metrics, defect reports, coverage summaries, and quality dashboards.
Experience and Qualifications
- 4–10 years of experience in Data Warehouse / Data Engineering QA.
- Strong hands‑on expertise with ETL tools such as ADF, SSIS, Informatica, and Databricks.
- Expert‑level SQL for complex data validation and reconciliation.
- Strong understanding of data warehousing concepts (SCDs, facts/dimensions, star schema).
- Strong analytical, debugging, and problem‑solving skills.
Preferred Skills
- Quality management and defect lifecycle management.
- Integration of automated tests into CI/CD pipelines using Azure DevOps.