Lead Data Engineer - Python/PySpark

Lead Data Engineer

We are looking for an experienced Lead Data Engineer to drive the architecture, development, and delivery of scalable enterprise data platforms and solutions.

Key Responsibilities :

- Lead the design, development, and implementation of scalable enterprise data architectures, data lakes, and data warehouse solutions.

- Build, maintain, and optimize batch and real-time data pipelines using Python, PySpark, Apache Spark, and SQL.

- Design and implement data ingestion, transformation, processing, and integration workflows from multiple structured and unstructured data sources.

- Establish efficient data processing frameworks and ensure data quality, consistency, availability, and reliability across platforms.

- Design and implement data models, data transformation processes, and storage solutions to support business intelligence and analytical requirements.

- Define and implement data engineering best practices, coding standards, architecture guidelines, and development processes.

- Lead technical design discussions, review solution approaches, and ensure alignment with enterprise architecture standards.

- Optimize data pipelines and distributed processing workloads to improve performance, scalability, and operational efficiency.

- Implement data governance practices, access controls, data security standards, and monitoring mechanisms.

- Work with cloud platforms and data services to design and manage reliable, scalable, and cost-effective data solutions.

- Collaborate with product owners, architects, business stakeholders, and cross-functional teams to understand requirements and translate them into technical solutions.

- Conduct code reviews, troubleshoot complex technical issues, and guide teams in resolving development and performance challenges.

- Mentor and support data engineers, encourage knowledge sharing, and drive continuous improvement within the engineering team.

- Monitor data platform performance, identify bottlenecks, and implement improvements to ensure system reliability and availability.

- Take ownership of technical delivery, ensure adherence to project timelines, and maintain the quality of engineering deliverables.

Required Skills and Qualifications :

- 8+ years of overall experience in Data Engineering, data platform development, and enterprise data solutions.

- Minimum 4+ years of hands-on experience with Python programming and development of data engineering applications.

- Strong expertise in Python, PySpark, Apache Spark, and complex SQL queries.

- Extensive experience in building and managing large-scale batch and real-time data pipelines.

- Strong understanding of data lakes, data warehouses, data modelling, data ingestion, and data transformation concepts.

- Experience working with distributed data processing frameworks and large-scale data environments.

- Good understanding of cloud-based data platforms and services across AWS, Azure, or Google Cloud.

- Experience with data integration, orchestration, automation, and workflow management tools.

- Strong knowledge of data governance, data quality, security, access management, and monitoring practices.

- Experience in designing scalable and high-performance data architectures.

- Familiarity with infrastructure automation, CI/CD pipelines, version control, and development best practices.

- Proven experience in technical leadership, architecture design, code reviews, and mentoring data engineering teams.

- Excellent communication and problem-solving skills, with the ability to explain technical concepts to non-technical stakeholders.

Ideal Candidate :

- The ideal candidate will be a hands-on technical leader with strong data engineering expertise, experience in building enterprise-scale data platforms, and the ability to guide engineering teams.

- They should demonstrate ownership of technical delivery, strong architectural thinking, performance optimization capabilities, and the ability to collaborate effectively with business and technical stakeholders.