AWS Big Data Platform Engineer (Netezza to AWS Migration)
Summary
Designs and migrates on-premises big data platforms (Netezza) to AWS, building scalable pipelines, ETL/ELT workflows, and real-time data processing systems using AWS services like EMR, Redshift, and Airflow.
Role: AWS Big Data Platform
Engineer (Netezza to AWS Migration)
Hybrid: 2 days a week in office
Job Summary: We are seeking a highly skilled Senior Platform Engineer with extensive
experience in Big Data technologies, AWS Cloud, and data platform engineering.
The ideal candidate will have strong expertise in designing, developing, and
supporting scalable data pipelines and distributed systems, along with hands-on
experience in Big Data ecosystem tools, AWS services, and real-time data
processing. This role involves working on data platform modernization, cloud
migrations, Data Warehousing and ETL in a fast-paced enterprise environment.
Key
Responsibilities
· Design, develop, maintain and support scalable
data pipelines using Big Data and AWS technologies.
· Lead and support data platform migration
initiatives (On-Prem to AWS Cloud), ideally with Netezza background.
· Develop and manage ETL/ELT processes using tools
like SSIS, Pentaho, or similar.
· Implement and manage AWS services such as EMR,
S3, EC2, Redshift, Glue, and Airflow.
· Build and optimize data workflows and
orchestration pipelines using Airflow.
· Work with real-time streaming technologies such
as Kafka and Spark Streaming.
· Perform data ingestion, transformation, and
validation from multiple data sources.
· Optimize SQL queries for performance and
scalability.
· Monitor system performance, troubleshoot issues,
and ensure system reliability.
· Collaborate with cross-functional teams
including developers, architects, and infrastructure teams.
Required
Technical Skills
· Cloud Technologies
· Strong experience in AWS Cloud services:
· EMR, EC2, S3, VPC, RDS, Redshift
· AWS Glue, IAM, CloudWatch, CloudFormation
· Airflow (or AWS Managed Workflows)
· Databases
Experience
working with:
· Netezza is mandatory
· SQL-based systems: SQL Server and PostgreSQL
· Data warehouses: Teradata, Redshift, Netezza
· ETL Tools
Hands-on
experience with:
· SSIS
· Programming & Scripting
Strong
proficiency in:
· SQL
· Shell scripting
Good to
have:
· Python
· Operating Systems
· Strong experience in Unix/Linux environments
Key
Qualifications
· 10+ years of experience in Data Engineering /
Big Data / Platform Engineering.
· Hands-on experience in AWS and Hadoop ecosystems.
· Experience in data migration projects (e.g.,
Hadoop to AWS, DB migrations).
· Strong understanding of data warehousing
concepts and ETL design patterns.
· Expertise in query optimization and performance
tuning.
· Experience supporting production data platforms.
· Bachelor’s degree in Computer Science,
Engineering, or related field.
Preferred
Qualifications
· Experience in Banking / Financial Services
domain
· Experience with data lake architecture and
governance
· Knowledge of DevOps tools and CI/CD pipelines
· Experience in platform engineering and framework
development
Soft Skills
· Excellent problem-solving and analytical skills
· Strong communication and collaboration abilities
· Ability to work independently and lead
initiatives
· Experience mentoring team members
Nice-to-Have
· Experience with tools like Ansible, Git, JIRA
· Exposure to NoSQL databases (HBase, MongoDB)
· Experience in data masking and security frame