Data Engineer (Ref 26540)
Summary
Design and maintain data pipelines, ETL processes, and dashboards using SQL, Python, and Databricks to ensure data quality and compliance with government standards.
Responsibilities
• Responsible for designing, developing, and maintaining data solutions for data generation, collection, and processing.
• Creating data pipelines, ensuring data quality, and implementing ETL processes to migrate and deploy data across systems.
• You are expected to be a subject matter expert, collaborate and manage the team to perform effectively.
• Engage with multiple teams, and contribute to key decisions while providing solutions to problems for your immediate team and across multiple teams.
• Query, clean, and transform datasets using SQL and Python on Databricks / Pyspark .
• Ensure data quality and consistency across systems in accordance with government Instruction Manual (IM8) standards.
• Build, enhance, and maintain dashboards for data monitoring, performance reporting, and user behaviour analytics.
• Partner with business stakeholders to understand data requirements and translate them into impactful visual insights.
• Documentation & Knowledge Management
• Develop clear and comprehensive user guides for dashboards, charts, and data workflows in Databricks
• Maintain technical documentation (e.g., on Confluence) for analytics processes, standards, and best practices.
Requirements
• Bachelor's degree in Computer Science, Engineering, or related field.
• At least 4-5 years of relevant working experience
• Proficiency in SQL and Python for data processing and automation.
• Hands-on experience in dashboard tools such as Tableau, Power BI, or equivalent visualisation tools.
• Familiarity with Databricks or similar big data/analytics platforms.
• Ability to produce high-quality technical documentation.
• Experience in system testing, SIT/UAT, or quality assurance processes.
• Knowledge of Spark for distributed data processing.
• Understanding of Git or other version control systems.
• Advanced proficiency in Data Engineering is required.
• Advanced proficiency in Data Access is recommended.
• Advanced proficiency in Data Migrations, Data Pipelines, and Data Processing is suggested.
Licence no: 12C6060