Lead Data Engineer
Posted Updated
Description
We are looking for a highly skilled Lead Data Engineer / Architect to design, build, and optimize scalable data pipelines and modern data platforms. This role is critical in transforming raw data into meaningful insights and enabling enterprise-level analytics.
Key Responsibilities
- Design, develop, and optimize ETL/ELT pipelines using Python, PySpark, Scala, and SQL
- Build and manage data workflows using Informatica, Databricks, AWS Glue, Snowflake
- Work across multi-cloud platforms (AWS, Azure, GCP, Microsoft Fabric)
- Manage and optimize databases such as Redshift, SQL Server, AWS RDS
- Orchestrate workflows using Apache Airflow
- Implement real-time streaming solutions using Apache Kafka
- Deploy and manage containerized applications using Kubernetes
- Automate processes using Unix shell scripting
- Implement CI/CD pipelines and ensure data governance
- Collaborate with BI teams for Tableau, Looker, Power BI
- Troubleshoot and optimize performance of pipelines and queries
- Maintain technical documentation
Requirements
- Bachelor’s/Master’s degree in Computer Science or related field
- 7+ years of experience in Data Engineering
- Strong skills in Python, PySpark, Scala, SQL
- Experience with Informatica, Databricks, AWS Glue
- Expertise in Snowflake & Redshift
- Experience in AWS / Azure / GCP
- Experience with Apache Airflow
- Knowledge of Apache Kafka & Kubernetes
- Experience with CI/CD pipelines
- Exposure to BI tools (Tableau, Looker, Power BI)