Senior Data Engineer ( DevOps)
Posted 2
views
Summary
Senior Data Engineer with a DevOps focus in Kuala Lumpur: building and supporting upstream subsurface data management apps, migrating legacy data to modern warehouses, and contributing to the OSDU open-source data platform. Core stack is Python, SQL, Azure, Terraform IaC, and CI/CD pipeline automation.
Experienced professional with strong expertise in back-end development and data analytics. Proficient in Python, and SQL, with hands-on experience in cloud-native development on Azure and Infrastructure as Code (IaC) using Terraform Enterprise.
Well-versed in designing and implementing CI/CD pipelines to automate deployment of both infrastructure and application code.
Role Overview:
- Develop and support Upstream Subsurface data management applications, including ingestion pipelines and workflows.
- Migrate data from legacy systems to modern data warehouses, ensuring data integrity and optimizing workflows throughout the process.
- Contribute to the Open Subsurface Data Universe (OSDU) community by building open-source data platforms and defining industry data standards.
Key Responsibilities
- Advocate and model secure coding practices.
- Promote quality development principles and techniques.
- Design and implement data pipelines for application and infrastructure deployment.
- Analyze gaps between legacy and target systems, identify missing data, and recommend improvements for completeness.
- Mentor others through teaching, coaching, and providing feedback.
- Develop deep expertise in relevant technology stacks through diverse development experiences.
Requirements:
- Degree in Computer Science, Engineering, or Data Engineering.
- Proficiency in Python and SQL scripting, experience with Oracle databases, APIs and Postman.
- Familiarity with DevOps tools (GitHub/GitLab, CI/CD) and cloud infrastructure (Azure, OpenShift, Airflow).
- Data engineering experience with Snowflake and Azure Data Factory is a plus.
- Strong communication skills for collaborating with team members, stakeholders, and external partners.
- Growth mindset with a proven track record of self-learning and continuous skill development.
- Experience with Power BI and Power Apps is an advantage.