Lead Data Engineer

We are seeking an experienced and highly skilled Lead Data Engineer to join our growing team. The ideal candidate will be responsible for designing, developing, and managing scalable cloud-based data platforms while leading a team of Data Engineers. This role requires strong expertise in Data Engineering, AWS cloud technologies, PySpark, Airflow, and Big Data ecosystems, along with the ability to deliver robust data solutions for BFSI (Banking, Financial Services, and Insurance) clients.

Responsibilities:

• Lead, mentor, and manage a team of Data Engineers, ensuring high-quality project delivery.
• Design, develop, and maintain scalable data pipelines, ETL/ELT workflows, and data integration solutions on AWS.
• Build and optimise Data Lake and Data Warehouse architectures to support analytical and operational workloads.
• Collaborate with business stakeholders, architects, data scientists, and analysts to understand data requirements and translate them into scalable solutions.
• Develop efficient data models and processing frameworks for large-scale data platforms.
• Ensure data quality, reliability, governance, and security across all data assets.
• Implement data security controls and compliance measures to protect sensitive BFSI data.
• Optimise data storage, processing, and retrieval for performance and cost efficiency.
• Develop and maintain APIs for data access and integration.
• Troubleshoot and resolve data-related issues, ensuring high system availability and reliability.
• Drive best practices in Data Engineering, DevOps, cloud architecture, and software development.
• Manage project timelines, task allocation, and successful delivery of data engineering initiatives.
• Stay current with emerging technologies and industry trends in Data Engineering and cloud platforms.

Requirements:

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• 6+ years of experience in Data Engineering with strong hands-on expertise.
• Proven experience leading and mentoring Data Engineering teams.
• Strong hands-on experience with: PySpark, Apache Airflow, AWS Cloud Services, Python, SQL.
• Strong understanding of: Data Warehousing, Data Lakes, ETL/ELT Frameworks, OLTP vs OLAP Architectures.
• Experience building and managing large-scale data pipelines and distributed data processing systems.
• Experience with Big Data technologies such as Apache Spark and Hadoop.
• Experience in API development, preferably using Python.
• Familiarity with Docker and Kubernetes.
• Experience with modern table formats such as Hudi, Iceberg, or Delta Lake.
• Strong problem-solving, analytical, and communication skills.
• Ability to work effectively in a collaborative, cross-functional environment.

Preferred Qualifications:

• Master's degree in Computer Science, Engineering, or a related discipline.
• Experience working in BFSI, Banking, Financial Services, Insurance, or FinTech domains.
• Knowledge of BFSI data governance, security standards, and regulatory compliance requirements.
• Exposure to multiple cloud platforms is an added advantage.
• Relevant cloud or data engineering certifications.
• 8 to 10 years.