Data Engineer
Posted Updated
Summary
Data Engineer in Kuala Lumpur who designs, builds, and maintains ETL/ELT pipelines on Microsoft Azure (Data Factory, Databricks, Azure SQL), optimizes SQL performance, and implements CI/CD with Azure DevOps, while coordinating with internal and vendor teams in an Agile setup.
- Design, develop, and maintain ETL/ELT pipelines on Microsoft Azure (Azure Data Factory, Databricks, Azure SQL / Azure SQL Managed Instance).
- Optimize SQL queries and stored procedures for performance, scalability, and reliability.
- Collaborate with data modelers to understand data structures, normalization, and data quality requirements.
- Implement CI/CD pipelines and deployment strategies using Azure DevOps (Repos, Pipelines, Release Management).
- Monitor, troubleshoot, and improve data pipelines; resolve data quality issues and incidents.
- Support the Data Tech Lead and internal teams with technical discussions and development work.
- Coordinate with Group Office tech team and local IT teams (architects, business analysts, etc.).
- Participate in Agile ceremonies and use tools like JIRA, Confluence, and Kanban boards.
Job Requirements:
- Minimum 3 years of hands-on experience in ETL/ELT, Databricks, SQL, and stored procedures.
- Strong skills in SQL tuning and performance optimization.
- Experience with Azure Data Factory and Databricks.
- Familiarity with Azure SQL or Azure SQL Managed Instance.
- Working knowledge of Azure DevOps (Repos, CI/CD pipelines).
- Understanding of data modeling and normalization principles.
- Experience working in Agile environments using tools like JIRA and Confluence.
- Experience in vendor management, including coordinating with external vendors, managing deliverables, and tracking SLAs.