AWS Data Engineer
Summary
Build and maintain AWS-based data pipelines and ETL workflows using Glue, S3, Lambda, and PySpark to process large datasets and migrate to Databricks.
- Design, develop, and maintain data pipelines and ETL workflows using AWS services.
- Implement orchestration and automation for data workflows.
- Work with large datasets to ensure data integrity, scalability, and performance.
- Collaborate with stakeholders to understand data requirements and deliver solutions.
- Deploy changes directly to production environments with confidence and accountability.
- Support migration efforts to Databricks and optimize workflows for performance
Qualifications
- 7 to 10 years of experience and above
- Experience with data lake architectures, big data technologies, and data pipeline orchestration.
- Familiarity with CI/CD practices for data engineering.
- AWS Certification (e.g., AWS Certified Data Analytics – Specialty or Solutions Architect) is a plus.
- Strong problem-solving skills and attention to detail.
Key Skills and Requirements
- Expert‑level fluency in AWS services relevant to data engineering, including Glue, S3, Lambda, Step Functions, and other related services.
- Strong proficiency in PySpark for distributed data processing.
- Advanced SQL skills for querying and optimizing data operations.
- Comfortable with pushing changes live without formal review, ensuring quality through self‑validation.
- Databricks experience is a plus.
- Ability to work independently and manage tasks in a siloed environment.
- Currently based in Kuala Lumpur, Malaysia