Data Engineer (Public Sector)
Our client is a statutory body in Hong Kong. They are looking for experienced talent to implement a scalable cloud-based data platform to enable sustainable, reusable, secure, efficient ingestion, transformation, and storage of enterprise data across multiple sources. Th individual should possess knowledge and experience of near real-time analytics capabilities to support business intelligence, predictive modelling, and faster decision-making for strategic initiatives.
Responsibilities
Optimize, operate at scale and enhance cloud-based data platform on Microsoft Azure (incl DataBricks)
Involve in cloud data platform enhancement project lifecycle with external vendor including design, development, testing, deployment, and documentation.
Work with Data Analyst to perform data preparation, cleansing, build optimized data models and visualization in Power BI
Perform proof of concept for cloud data products
Validate data workflows and integration points through rigorous testing (e.g., pipeline validation, schema checks, and performance benchmarking) to ensure solutions meet business and technical requirements.
Create operational playbooks and automation scripts to streamline deployment, monitoring, and troubleshooting, enabling efficient handover and long-term maintainability of data solutions.
Document data pipeline architecture, deployment processes, and operational runbooks to support effective troubleshooting and post-go-live maintenance by internal teams
Requirements:
Bachelor’s degree in computer science, Information Technology, Data Science or related field
3 years of professional experience related to data platform
Experience in Python, SQL, Spark, Azure Data Factory, Event Hub / Kafka / Event Messaging, Azure Data Lake Storage Gen2, Databricks, Unity Catalog, Power BI
Understanding of data lifecycle and governance principles (e.g., data quality, lineage, security, and compliance
Experience in software development lifecycle (SDLC)