Aws Data Engineer

About the Role

We are seeking a highly skilled AWS Data Engineer with strong expertise in PySpark, SQL, and AWS Redshift to design, build, and optimize scalable data pipelines and cloud-based data platforms. The ideal candidate should have hands-on experience with AWS data services, large-scale data processing, ETL development, and data warehousing solutions.

The role involves working closely with data architects, data scientists, analysts, and business stakeholders to deliver reliable, high-performance data solutions supporting analytics, reporting, and advanced AI/ML initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and AWS services.
• Build and optimize ETL/ELT workflows for structured and unstructured data.
• Develop and manage enterprise data warehouses using AWS Redshift.
• Write complex and optimized SQL queries for data extraction, transformation, and analysis.
• Ingest data from APIs, databases, flat files, and streaming platforms.
• Perform data cleansing, transformation, validation, and quality checks.
• Optimize Redshift schemas, tables, and queries for performance and scalability.
• Implement data governance, security, and compliance best practices.
• Monitor, troubleshoot, and resolve data pipeline and performance issues.
• Collaborate with cross-functional teams and maintain technical documentation.
• Support production deployments and ongoing data platform operations.

Required Technical Skills

Core Data Engineering

• Strong hands-on experience with PySpark.
• Advanced SQL development and performance optimization.
• ETL/ELT pipeline development.
• Data modeling and data warehousing concepts.
• Working knowledge of distributed data processing frameworks.

AWS Services

• AWS Redshift Mandatory
• AWS S3 and AWS Glue.
• AWS Lambda and AWS IAM.
• AWS CloudWatch and AWS Step Functions.
• AWS Secrets Manager.

Big Data & Processing

• Apache Spark and PySpark.
• Data partitioning and performance tuning.
• Batch and near real-time data processing.
• Data quality and validation frameworks.

Database Technologies

• Amazon Redshift, PostgreSQL, and MySQL.
• SQL Server is good to have.

DevOps & Automation

• Git, GitHub, or Bitbucket.
• CI/CD pipelines.
• Infrastructure and deployment automation.
• Shell and Python scripting.

Preferred Skills

• Experience with Data Lakes and Lakehouse architectures.
• Exposure to Databricks or Amazon EMR.
• Knowledge of Apache Airflow or AWS Managed Workflows for Apache Airflow (MWAA).
• Experience with Kafka or real-time streaming platforms.
• Familiarity with Terraform, CloudFormation, or Infrastructure as Code.
• Exposure to Snowflake or other cloud data warehouse platforms.
• Knowledge of data governance and metadata management tools.
• Experience supporting Analytics, BI, or Machine Learning workloads.