Data Engineer

Job Summary

We are looking for an experienced Data Engineer with strong hands-on expertise in Azure/AWS Databricks, PySpark, Python, SQL, and Apache Airflow. The candidate will be responsible for understanding existing data pipelines, analyzing business and technical logic, implementing enhancements, troubleshooting issues, and supporting end-to-end data engineering activities.

Key Responsibilities

• Understand and analyze existing data engineering pipelines, codebases, and business logic.
• Develop, maintain, and enhance data pipelines using Databricks, PySpark, Python, and SQL.
• Work with Apache Airflow for workflow orchestration, scheduling, and monitoring.
• Perform impact analysis before implementing code changes or enhancements.
• Troubleshoot pipeline failures, resolve data processing issues, and perform root cause analysis.
• Understand upstream and downstream dependencies to ensure smooth data flow.
• Optimize data processing jobs and SQL queries for performance and scalability.
• Perform data validation and ensure data quality and consistency.
• Support end-to-end pipeline development, maintenance, and production issues.
• Collaborate with cross-functional teams to deliver reliable data engineering solutions.

Mandatory Skills

• Strong hands-on experience with Databricks on Azure or AWS.
• Proficiency in PySpark and Python.
• Strong SQL skills, including query writing and optimization.
• Experience with Apache Airflow for pipeline orchestration.
• Experience maintaining and enhancing existing data pipelines.
• Ability to analyze existing code, understand dependencies, and implement changes safely.
• Strong troubleshooting and problem-solving skills.

Preferred Qualifications

• Experience with cloud-based data engineering solutions on Azure or AWS.
• Knowledge of ETL/ELT processes and data pipeline architecture.
• Experience in production support, performance tuning, and data quality validation.