Data Engineer (ETL, Snowflake, AWS Glue)
Posted 3
views
Summary
Data engineer in Singapore who builds ETL pipelines and data lake feeds, migrates legacy Oracle/Hive transformation jobs to Snowflake, Spark and AWS Glue, and leads a team running BAU operations and production support. Core stack: AWS (S3, Glue, Airflow/MWAA, Lambda), Snowflake, SQL, Python/PySpark.
Roles & Responsibilities:
. Develop tools to improve data flows between internal/external systems and the data lake/warehouse.
. Work with stakeholders to understand needs for data structure, availability, scalability, and accessibility.
. Build robust and reproducible data ingest pipelines to collect, clean, harmonize, merge, and consolidate data sources.
. Understanding existing data applications and infrastructure architecture
. Build and support new data feeds for various Data Management layers and Data Lakes
. Evaluate business needs and requirements.
. Support migration of existing data transformation jobs in Oracle, and MS-SQL to Snowflake.
. Lead the migration of the existing data transformation jobs in Oracle, Hive, Impala etc. into Spark, Python on Glue etc.
. Able to document the processes and steps.
. Develop and maintain datasets.
. Improve data quality and efficiency.
. Lead Business requirements and deliver accordingly.
. Collaborate with Data Scientists, Architect and Team on several Data Analytics projects.
. Collaborate with DevOps Engineer to improve system deployment and monitoring process.
. Experience in critical production support and how the BAU functions is preferred.
Key Requirements:
. Lead a team of data engineers in managing day-to-day BAU operations, production support, incident management, and platform stability. . Strong AWS knowledge in terms of designing new architecture and providing optimized solutions for existing ones. (S3, Glue, DMS, MWAA, AMS, IAM). . In-depth knowledge with respect to Snowflake and its architecture. . Prefer prior Experience in BAU environment. . Good knowledge on Airflow and MWAA. . Hands-on experience in SQL/Python/Pyspark. . Expertise in optimizing techniques in cloud environments. . Should have the vision on data strategy and able to deliver the same. Must Have: AWS - Glue, Airflow, Lambda, S3 , Snowflake (Related Cloud DB) SQL ,Python/ Spark,CI/CD Good To have : ECS,GIT, Oracle RajkumarRenganathan
Quess Selection& Services, Singapore EA Licence Number: 23C2060 Registration ID is R1983662 Disclaimer: The company is committed to ensuring the privacy and security of your information. By submitting this form, you consent to the collection, processing, and retention of the information you provide. The data collected (which may include your contact details, educational background, work experience and skills) will be used solely for the purpose of evaluating your qualifications for the position you're applying for. Your data will be stored securely and retained for the duration necessary to fulfill our hiring process. If you are not selected for the position, your data will be kept on file for a limited period in case future opportunities arise. You have the right to access, correct, or delete your data at any time by contacting us at Quess Singapore | A Leading Staffing Services Provider in Singapore (quesscorp.sg)
. Lead a team of data engineers in managing day-to-day BAU operations, production support, incident management, and platform stability. . Strong AWS knowledge in terms of designing new architecture and providing optimized solutions for existing ones. (S3, Glue, DMS, MWAA, AMS, IAM). . In-depth knowledge with respect to Snowflake and its architecture. . Prefer prior Experience in BAU environment. . Good knowledge on Airflow and MWAA. . Hands-on experience in SQL/Python/Pyspark. . Expertise in optimizing techniques in cloud environments. . Should have the vision on data strategy and able to deliver the same. Must Have: AWS - Glue, Airflow, Lambda, S3 , Snowflake (Related Cloud DB) SQL ,Python/ Spark,CI/CD Good To have : ECS,GIT, Oracle RajkumarRenganathan
Quess Selection& Services, Singapore EA Licence Number: 23C2060 Registration ID is R1983662 Disclaimer: The company is committed to ensuring the privacy and security of your information. By submitting this form, you consent to the collection, processing, and retention of the information you provide. The data collected (which may include your contact details, educational background, work experience and skills) will be used solely for the purpose of evaluating your qualifications for the position you're applying for. Your data will be stored securely and retained for the duration necessary to fulfill our hiring process. If you are not selected for the position, your data will be kept on file for a limited period in case future opportunities arise. You have the right to access, correct, or delete your data at any time by contacting us at Quess Singapore | A Leading Staffing Services Provider in Singapore (quesscorp.sg)