Senior Data Engineer- Malaysia
Posted 6
views
Summary
A senior data engineer designs, builds, and maintains AWS-based data pipelines and ETL workflows, deploying changes directly to production and supporting a migration to Databricks. Core tech includes AWS (Glue, S3, Lambda, Step Functions), PySpark, and advanced SQL.
- Design, develop, and maintain data pipelines and ETL workflows using AWS services.
- Implement orchestration and automation for data workflows.
- Work with large datasets to ensure data integrity, scalability, and performance.
- Collaborate with stakeholders to understand data requirements and deliver solutions.
- Deploy changes directly to production environments with confidence and accountability.
- Support migration efforts to Databricks and optimize workflows for performance
Qualifications
- 7 to 10 years of experience and above
- Experience with data lake architectures, big data technologies, and data pipeline orchestration.
- Familiarity with CI/CD practices for data engineering.
- AWS Certification (e.g., AWS Certified Data Analytics – Specialty or Solutions Architect) is a plus.
- Strong problem-solving skills and attention to detail.
Key Skills and Requirements
- Expert-level fluency in AWS services relevant to data engineering, including Glue, S3, Lambda, Step Functions, and other related services.
- Strong proficiency in PySpark for distributed data processing.
- Advanced SQL skills for querying and optimizing data operations.
- Comfortable with pushing changes live without formal review, ensuring quality through self-validation.
- Databricks experience is a plus.
- Ability to work independently and manage tasks in a siloed environment.
- Currently based in Kuala Lumpur, Malaysia