Data Engineer

Lead Data Engineer

Skills: Python, PySpark, Databricks, SQL, AI/ML

Job Title: Lead Data Engineer Python / PySpark / Databricks / AI/ML

Job Summary

We are looking for an experienced Lead Data Engineer with strong expertise in Python, PySpark, Databricks, and SQL to design, develop, and optimize scalable data engineering solutions. The candidate will lead data engineering initiatives, build robust ETL/ELT pipelines, implement modern lakehouse architectures, and support AI/ML use cases by delivering high-quality, reliable, and analytics-ready data.

The ideal candidate should have strong technical leadership skills, experience with large-scale data processing, and the ability to collaborate with data scientists, analysts, architects, and business stakeholders.

Key Responsibilities

• Lead the design, development, and implementation of scalable data pipelines using Python, PySpark, and Databricks.
• Design and develop ETL/ELT workflows to process structured, semi-structured, and unstructured data.
• Build and maintain data lakehouse solutions using Databricks and Delta Lake.
• Develop complex SQL queries, stored procedures, and performance-optimized data transformations.
• Optimize Spark jobs, including partitioning, caching, joins, and cluster configuration, to improve performance and reduce costs.
• Implement data ingestion and transformation pipelines for batch and streaming data.
• Develop data quality checks, reconciliation processes, monitoring, and error-handling mechanisms.
• Collaborate with AI/ML teams to prepare, transform, and manage datasets for machine learning and advanced analytics.
• Support ML workflows, feature engineering, and integration of data pipelines with machine learning platforms.
• Implement CI/CD, version control, automated testing, and deployment practices for data engineering solutions.
• Ensure data security, governance, access control, and compliance with organizational standards.
• Mentor junior data engineers, conduct code reviews, and establish engineering best practices.
• Work closely with business stakeholders to translate requirements into scalable technical solutions.

Required Technical Skills

1. Python

• Strong programming skills in Python.
• Experience with Pandas, NumPy, and data processing libraries.
• Ability to build reusable modules, frameworks, and automation scripts.
• Knowledge of exception handling, logging, testing, and performance optimization.

2. PySpark / Apache Spark

• Strong hands-on experience with PySpark and distributed data processing.
• Expertise in Spark DataFrames, transformations, actions, and Spark SQL.
• Understanding of partitioning, shuffling, caching, broadcast joins, and execution plans.
• Experience troubleshooting and optimizing large-scale Spark workloads.

3. Databricks

• Hands-on experience with Databricks notebooks, clusters, jobs, and workflows.
• Experience with Delta Lake, Delta tables, and lakehouse architecture.
• Knowledge of Unity Catalog, access controls, and data governance.
• Experience with Delta Live Tables / Lakeflow Declarative Pipelines, where applicable.
• Familiarity with Databricks Workflows and deployment automation.

4. SQL

• Strong expertise in SQL, including complex joins, CTEs, subqueries, and window functions.
• Experience in query tuning and execution-plan analysis.
• Knowledge of relational databases, data modeling, and dimensional modeling.
• Experience developing efficient queries for large datasets.

5. AI/ML and Data Engineering Integration

• Understanding of machine learning concepts and the data lifecycle.
• Experience with feature engineering and preparation of ML-ready datasets.
• Familiarity with MLflow and model lifecycle management.
• Knowledge of feature stores, model-serving integrations, or vector search is an advantage.
• Experience supporting generative AI or RAG data pipelines is a plus.

Additional Preferred Skills

• Experience with cloud platforms such as Azure, AWS, or Google Cloud.
• Knowledge of Azure Data Lake Storage, Amazon S3, or Google Cloud Storage.
• Experience with orchestration tools such as Airflow or cloud-native workflow services.
• Familiarity with Git, CI/CD pipelines, Docker, and automated testing.
• Understanding of data warehousing, medallion architecture, and data governance.
• Experience working in Agile/Scrum environments.

Leadership Responsibilities

• Lead technical design discussions and architecture reviews.
• Define coding standards and reusable data engineering patterns.
• Review code and ensure performance, scalability, reliability, and maintainability.
• Mentor team members and support technical problem-solving.
• Coordinate with architects, product owners, data scientists, and business teams.
• Drive continuous improvement in data platform performance and engineering practices.

Experience and Qualifications

• Experience: Typically 812+ years in data engineering, with substantial hands-on experience in Python, PySpark, and Databricks.
• Leadership: Demonstrated experience leading technical projects or mentoring data engineering teams.
• Education: Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience.

Key Competencies

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder-management abilities.
• Ability to design scalable and cost-effective data solutions.
• Strong ownership, technical decision-making, and mentoring skills.

Expected Deliverables

• Scalable and reusable ETL/ELT pipelines.
• Optimized Databricks and PySpark workloads.
• Reliable, governed, high-quality data assets.
• Analytics-ready and ML-ready datasets.
• Well-documented, tested, and production-ready data engineering solutions.