Data Engineer
Posted 2
views
Summary
Data Engineer at a property development group in Kuala Lumpur who builds and maintains data pipelines into the company's data lake/warehouse, develops web scraping scripts, automates deployment via CI/CD, and ensures data quality and security. Core stack includes Azure (Functions, Data Factory, Databricks), AWS (Glue, Lambda, Redshift), and Python/Java/Scala with Spark or Hadoop.
Job Description:
- Create and manage data pipelines for diverse sources in our data lake/warehouse.
- Collaborate with analysts to implement necessary data transformations.Research relevant data sources for property development.
- Develop web scraping scripts for data acquisition.
- Automate deployment and testing of data pipelines using CI/CD.
- Utilize Azure, AWS, and other technologies for data solutions.
- Implement data integration between systems.
- Monitor and optimize pipeline performance.
- Ensure data security, compliance, and quality.
- Stay updated on data engineering trends.
Requirements
- Bachelor's or Master's degree in Computer Science, Engineering, or related field.
- Proven experience as a Data Engineer, with strong understanding of data engineering concepts.
- Hands-on experience with Azure (Azure Function, Azure Data Factory, Azure Databricks) and/or AWS (AWS Glue, Lambda, Amazon Redshift).
- Proficiency in Python, Java, or Scala, and experience with data processing frameworks ("e.g., Spark, Hadoop").
- Knowledge of data modelling, warehousing, and database technologies.