Data Engineer
Summary
Data Engineer (5–10 yrs) in Bengaluru who designs and builds scalable Python/PySpark batch data pipelines, SQL transformations, and reusable ETL frameworks at Persistent. Day-to-day includes data-quality checks, Airflow orchestration, GitHub Actions/Harness CI/CD, code reviews, and troubleshooting production pipelines.
Key Responsibilities
Design and develop scalable Python and PySpark/Spark-based data pipelines for batch data processing and integration.
Build reusable frameworks and components for data ingestion, transformation, validation, and publishing.
Work with structured and semi-structured data from databases, files, APIs, and enterprise data platforms.
Develop and optimize complex SQL and Spark transformations for large-volume datasets.
Implement data-quality checks, exception handling, logging, reconciliation, and operational controls.
Write modular, maintainable, testable Python/PySpark code following enterprise engineering standards.
Use GitHub for source-code management, branching, pull requests, and collaborative development.
Work with GitHub Actions and/or Harness to support CI/CD workflows for building, testing, and deploying data applications.
Develop and maintain workflow orchestration using Apache Airflow or equivalent enterprise scheduling/orchestration platforms.
Troubleshoot pipeline failures, performance issues, data-quality problems, and production defects.
Participate in code reviews, unit/integration testing, release activities, and production support.
Collaborate with data architects, platform engineers, DevOps teams, product owners, and other engineering teams.
Required Technical Skills
Must Have
Strong hands-on programming experience with Python.
Strong experience with Apache Spark / PySpark.
Strong working knowledge of SQL and data transformation techniques.
Experience developing production-grade ETL/ELT and batch data pipelines.
Understanding of distributed data-processing concepts and Spark performance optimization.
Experience working with Git/GitHub, including branching, pull requests, and code reviews.
Practical exposure to CI/CD, preferably using GitHub Actions and/or Harness.
Experience with Apache Airflow or another scheduling/workflow orchestration tool.
Experience with automated testing, logging, error handling, deployment, and production troubleshooting.