Python Data Testing Lead

Role & responsibilities

• Lead end-to-end testing strategy for ETL pipelines, data warehouses, and big data platforms
• Design and implement scalable data validation frameworks
• Perform database testing across SQL and NoSQL systems
• Validate data across data lakes (e.g., Snowflake) and distributed systems
• Review and improve test code and automation frameworks
• Collaborate with data engineers to ensure data quality and pipeline reliability
• Build and maintain automated testing solutions for batch and streaming pipelines
• Optimize test processes for performance, scalability, and accuracy
• Integrate testing workflows into CI/CD pipelines
• Ensure compliance with data governance and quality standards.

Preferred candidate profile

1. Expertise in ETL testing and data validation across complex pipelines
2. Strong proficiency in Python programming
3. Hands-on experience with PySpark and/or Scala for large-scale data processing
4. Advanced knowledge of SQL, data modeling, and database testing (SQL & NoSQL)
5. Experience with data warehouses and data lakes (e.g., Snowflake)
6. Strong experience with Databricks
7. Deep understanding of Big Data ecosystems
8. Experience in testing data pipelines and ETL workflows
9. Strong understanding of data modeling techniques
10. Input/output validation for ML pipelines.

Good to Have Skills

• Experience with Apache Flink (streaming and batch processing)
• Exposure to CI/CD pipelines (Jenkins, GitLab, etc.)
• Knowledge of Docker and Kubernetes for containerized environments
• Working knowledge of Linux and shell scripting
• Hands-on experience with AWS backend services