Senior Data Quality Engineer
5+ years of professional experience in Data Quality, Data Engineering, or Big Data QA.
Strong Python & PySpark: Advanced hands-on experience building distributed validation jobs and testing scripts.
Database Proficiency: Solid expertise querying and validating structured (MySQL) and semistructured (MongoDB) stores.
Search Engine Knowledge: Experience validating high-throughput search indices in Elasticsearch. Architecture Understanding: Strong background in Data Warehouse and Data Lake design, multi-tenant schemas, and ETL/ELT pipelines.
Software Engineering: Clean code principles, version control (Git), automated testing frameworks, and CI/CD.
Requirements
Framework & Automated Pipeline Testing Data Quality Automation: Architect, implement, and extend automated testing frameworks (e.g., Great Expectations, Soda Core, or custom PySpark modules) within ETL/ELT pipelines.
Data Reconciliation: Build distributed reconciliation jobs in PySpark to verify end-to-end data consistency across MongoDB, MySQL, Elasticsearch, and the Data Lake.
Anomaly & Drift Detection: Implement real-time profiling to identify schema drift, unexpected NULL values, volume drops, and telemetry anomalies.
2. Data Observability & Monitoring
Alerting & SLAs: Define SLA/SLO metrics for data freshness, completeness, and accuracy; build real-time monitoring dashboards and alerting integrations.
Root Cause Analysis (RCA): Lead technical investigations into pipeline failures, duplicate telemetry records, or corrupted time-series data, implementing permanent fixes.
3. Engineering Standards & Governance Lineage & Documentation: Map data flows and maintain automated cataloging to ensure clear visibility from ingestion to analytics downstream.
CI/CD Integration: Embed data validation gates into continuous integration/deployment (CI/CD) pipelines to prevent bad code or schema changes from hitting production.
Cross-Team Collaboration: Partner closely with Data Engineering, Analytics, and IoT teams to align validation rules with business requirements.