Data engineer
Posted Updated
Summary
Validate and safeguard enterprise data platforms end to end: build and test large-scale pipelines with PySpark and Databricks, run SQL-based reconciliations and data quality checks, and automate quality controls with CI/CD and observability tooling across Azure, AWS, or GCP cloud Lakehouse environments.
Job Summary (List Format):
- Ensure accuracy, reliability, and scalability of enterprise data platforms as a Data Quality Engineer with strong Data Engineering skills.
- Perform end-to-end data pipeline validation, including ingestion, transformation, and consumption phases.
- Execute data quality checks, source-to-target reconciliations, and resolve data anomalies.
- Define and implement data quality frameworks, metrics, and controls.
- Develop and validate large-scale data pipelines using PySpark and Databricks.
- Work with Hadoop ecosystem components (Hive, HDFS, Spark) and Lakehouse environments.
- Support and optimize ETL/ELT workflows for data integrity and performance.
- Write advanced SQL queries for data profiling, reconciliation, and root cause analysis.
- Validate business rules and transformation logic using SQL (joins, window functions, CTEs, aggregations).
- Monitor and validate data in Databricks Lakehouse on cloud platforms (Azure, AWS, GCP).
- Collaborate with Data Engineers, Architects, and Analysts for reliable data delivery.
- Analyze and resolve production data issues; manage data defects and implement automated quality checks.
- Utilize automated testing frameworks, data observability tools, and CI/CD practices for data pipelines.
- Preferred experience with Delta Lake, Unity Catalog, Airflow, Kafka, or similar technologies.
- Understanding of data modeling concepts like Star Schema and Snowflake Schema.
- Ensure accuracy, reliability, and scalability of enterprise data platforms as a Data Quality Engineer with strong Data Engineering skills.
- Perform end-to-end data pipeline validation, including ingestion, transformation, and consumption phases.
- Execute data quality checks, source-to-target reconciliations, and resolve data anomalies.
- Define and implement data quality frameworks, metrics, and controls.
- Develop and validate large-scale data pipelines using PySpark and Databricks.
- Work with Hadoop ecosystem components (Hive, HDFS, Spark) and Lakehouse environments.
- Support and optimize ETL/ELT workflows for data integrity and performance.
- Write advanced SQL queries for data profiling, reconciliation, and root cause analysis.
- Validate business rules and transformation logic using SQL (joins, window functions, CTEs, aggregations).
- Monitor and validate data in Databricks Lakehouse on cloud platforms (Azure, AWS, GCP).
- Collaborate with Data Engineers, Architects, and Analysts for reliable data delivery.
- Analyze and resolve production data issues; manage data defects and implement automated quality checks.
- Utilize automated testing frameworks, data observability tools, and CI/CD practices for data pipelines.
- Preferred experience with Delta Lake, Unity Catalog, Airflow, Kafka, or similar technologies.
- Understanding of data modeling concepts like Star Schema and Snowflake Schema.