AWS Data Engineer
Posted 3
views
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Python and PySpark.
- Build and manage ETL/ELT workflows for processing structured and unstructured data.
- Develop and optimize Spark applications to handle large-scale datasets efficiently.
- Implement cloud-native data solutions using AWS services.
- Design and maintain Data Lakes on AWS.
- Perform data ingestion from multiple sources including APIs, databases, files, and streaming platforms.
- Monitor, troubleshoot, and optimize data workflows for performance and reliability.
- Ensure data quality, integrity, security, and governance standards are followed.
- Implement automation and orchestration of data workflows using scheduling tools.
- Collaborate with cross-functional teams for requirement gathering and solution design.
- Conduct code reviews, unit testing, and performance tuning.
- Prepare technical documentation and deployment procedures.
- Support production incidents and perform root cause analysis.
Mandatory Skills
- Strong expertise in Python
- Hands-on experience with PySpark
- Proficiency in SQL and query optimization