Data Engineer (Snowflake & AWS)
Summary
Lead cloud-based data engineering operations, build and optimize AWS and Snowflake pipelines, and manage production support for scalable data platforms.
Job Summary
We are looking for an experienced
Data BAU Lead - Data Engineering
to lead data operations, production support, and continuous improvement initiatives within a cloud-first data environment. The ideal candidate will have strong technical leadership capabilities, hands-on expertise in
AWS, Snowflake, Data Engineering, and BAU operations , with proven experience managing large-scale data platforms, production incidents, and data transformation initiatives.
The role requires a strong understanding of modern data architecture, cloud technologies, ETL pipelines, data lakes, data warehouses, and the ability to lead engineering teams while collaborating with business stakeholders, architects, data scientists, and DevOps teams.
Key Responsibilities
Data Engineering & Platform Management
Lead the Data Engineering BAU team responsible for daily operations, production support, incident management, and platform stability. Design, develop, and maintain scalable data pipelines and data processing solutions. Develop tools and frameworks to improve data movement between internal/external systems and enterprise data platforms. Build robust and reusable data ingestion pipelines to collect, cleanse, transform, harmonize, and consolidate data from multiple sources. Support and enhance existing data applications, infrastructure, and architecture. Develop and maintain datasets, data models, and data management processes. Improve data quality, reliability, performance, and operational efficiency.
Cloud & Data Platform Migration
Support migration of existing data transformation workloads from Oracle and MS SQL environments to Snowflake. Lead migration of legacy transformation processes from Oracle, Hive, and Impala into modern cloud-based solutions using Spark, Python, and AWS Glue. Design and optimize cloud-based data solutions using AWS services. Evaluate and recommend data technologies, platforms, and tools aligned with business strategy.
Production Support & BAU Operations
Manage critical production support activities, ensuring timely resolution of incidents and operational issues. Establish and improve BAU processes, monitoring, alerting, and operational controls. Collaborate with DevOps teams to enhance CI/CD deployment processes and system monitoring. Ensure proper documentation of processes, workflows, technical solutions, and operational procedures.
Stakeholder & Team Leadership
Work closely with business stakeholders to understand data requirements, availability, scalability, and accessibility needs. Lead requirement analysis and deliver effective data solutions. Collaborate with Data Scientists, Solution Architects, Engineers, and Business Teams on analytics initiatives. Mentor and guide data engineers, promoting coding best practices, design principles, and engineering excellence. Manage stakeholder communication including senior leadership updates when required.
Required Technical Skills
Cloud & Data Platforms
Strong hands-on experience with AWS Cloud Data Services, including: Amazon S3 AWS Glue AWS DMS AWS MWAA (Managed Workflows for Apache Airflow) IAM Amazon RDS Amazon Kinesis AWS Lambda AWS Step Functions Strong understanding of cloud architecture design and optimization techniques. Extensive knowledge of Snowflake architecture, performance optimization, and implementation practices. Experience working with enterprise-scale Data Lakes and Data Warehouses.
Data Engineering Skills
Strong programming experience in: SQL Python PySpark Unix/Linux Shell Scripting Strong experience with: ETL/ELT development Data ingestion frameworks Data integration Data transformation Data modelling Data quality management Good understanding of distributed systems and scalable data processing. Experience with data streaming technologies is preferred.
Workflow & DevOps
Strong knowledge of: Apache Airflow / AWS MWAA CI/CD pipelines Git-based deployments Agile development methodologies SDLC lifecycle
Experience & Qualification Requirements
8+ years of experience in Data Engineering. 5+ years of recent hands-on coding experience as a Lead Engineer managing production support and BAU operations. Minimum 2+ years of experience with large-scale datasets, Data Lakes, and Data Warehouse technologies. Strong preference for hands-on experience with Snowflake. Minimum 3+ years of experience with AWS Data Engineering services including AWS Glue, S3, RDS, Lambda, Kinesis, Step Functions, and Airflow. Bachelor's degree in Computer Science, Information Technology, Engineering, or related STEM discipline. Experience working in Agile, dynamic, customer-focused environments.
We are looking for an experienced
Data BAU Lead - Data Engineering
to lead data operations, production support, and continuous improvement initiatives within a cloud-first data environment. The ideal candidate will have strong technical leadership capabilities, hands-on expertise in
AWS, Snowflake, Data Engineering, and BAU operations , with proven experience managing large-scale data platforms, production incidents, and data transformation initiatives.
The role requires a strong understanding of modern data architecture, cloud technologies, ETL pipelines, data lakes, data warehouses, and the ability to lead engineering teams while collaborating with business stakeholders, architects, data scientists, and DevOps teams.
Key Responsibilities
Data Engineering & Platform Management
Lead the Data Engineering BAU team responsible for daily operations, production support, incident management, and platform stability. Design, develop, and maintain scalable data pipelines and data processing solutions. Develop tools and frameworks to improve data movement between internal/external systems and enterprise data platforms. Build robust and reusable data ingestion pipelines to collect, cleanse, transform, harmonize, and consolidate data from multiple sources. Support and enhance existing data applications, infrastructure, and architecture. Develop and maintain datasets, data models, and data management processes. Improve data quality, reliability, performance, and operational efficiency.
Cloud & Data Platform Migration
Support migration of existing data transformation workloads from Oracle and MS SQL environments to Snowflake. Lead migration of legacy transformation processes from Oracle, Hive, and Impala into modern cloud-based solutions using Spark, Python, and AWS Glue. Design and optimize cloud-based data solutions using AWS services. Evaluate and recommend data technologies, platforms, and tools aligned with business strategy.
Production Support & BAU Operations
Manage critical production support activities, ensuring timely resolution of incidents and operational issues. Establish and improve BAU processes, monitoring, alerting, and operational controls. Collaborate with DevOps teams to enhance CI/CD deployment processes and system monitoring. Ensure proper documentation of processes, workflows, technical solutions, and operational procedures.
Stakeholder & Team Leadership
Work closely with business stakeholders to understand data requirements, availability, scalability, and accessibility needs. Lead requirement analysis and deliver effective data solutions. Collaborate with Data Scientists, Solution Architects, Engineers, and Business Teams on analytics initiatives. Mentor and guide data engineers, promoting coding best practices, design principles, and engineering excellence. Manage stakeholder communication including senior leadership updates when required.
Required Technical Skills
Cloud & Data Platforms
Strong hands-on experience with AWS Cloud Data Services, including: Amazon S3 AWS Glue AWS DMS AWS MWAA (Managed Workflows for Apache Airflow) IAM Amazon RDS Amazon Kinesis AWS Lambda AWS Step Functions Strong understanding of cloud architecture design and optimization techniques. Extensive knowledge of Snowflake architecture, performance optimization, and implementation practices. Experience working with enterprise-scale Data Lakes and Data Warehouses.
Data Engineering Skills
Strong programming experience in: SQL Python PySpark Unix/Linux Shell Scripting Strong experience with: ETL/ELT development Data ingestion frameworks Data integration Data transformation Data modelling Data quality management Good understanding of distributed systems and scalable data processing. Experience with data streaming technologies is preferred.
Workflow & DevOps
Strong knowledge of: Apache Airflow / AWS MWAA CI/CD pipelines Git-based deployments Agile development methodologies SDLC lifecycle
Experience & Qualification Requirements
8+ years of experience in Data Engineering. 5+ years of recent hands-on coding experience as a Lead Engineer managing production support and BAU operations. Minimum 2+ years of experience with large-scale datasets, Data Lakes, and Data Warehouse technologies. Strong preference for hands-on experience with Snowflake. Minimum 3+ years of experience with AWS Data Engineering services including AWS Glue, S3, RDS, Lambda, Kinesis, Step Functions, and Airflow. Bachelor's degree in Computer Science, Information Technology, Engineering, or related STEM discipline. Experience working in Agile, dynamic, customer-focused environments.