Data Engineer (Ref 26540)
Summary
Designs and maintains data pipelines, ETL processes, and dashboards using SQL, Python, Databricks, and Spark to ensure data quality and insights for business stakeholders.
Responsibilities
- Responsible for designing, developing, and maintaining data solutions for data generation, collection, and processing.
- Creating data pipelines, ensuring data quality, and implementing ETL processes to migrate and deploy data across systems.
- You are expected to be a subject matter expert, collaborate and manage the team to perform effectively.
- Engage with multiple teams, and contribute to key decisions while providing solutions to problems for your immediate team and across multiple teams.
- Query, clean, and transform datasets using SQL and Python on Databricks / Pyspark .
- Ensure data quality and consistency across systems in accordance with government Instruction Manual (IM8) standards.
- Build, enhance, and maintain dashboards for data monitoring, performance reporting, and user behaviour analytics.
- Partner with business stakeholders to understand data requirements and translate them into impactful visual insights.
Documentation & Knowledge Management
- Develop clear and comprehensive user guides for dashboards, charts, and data workflows in Databricks
- Maintain technical documentation (e.g., on Confluence) for analytics processes, standards, and best practices.
Requirements
- Bachelor's degree in Computer Science, Engineering, or related field.
- At least 4-5 years of relevant working experience
- Proficiency in SQL and Python for data processing and automation.
- Hands-on experience in dashboard tools such as Tableau, Power BI, or equivalent visualisation tools.
- Familiarity with Databricks or similar big data/analytics platforms.
- Ability to produce high-quality technical documentation.
- Experience in system testing, SIT/UAT, or quality assurance processes.
- Knowledge of Spark for distributed data processing.
- Understanding of Git or other version control systems.
- Advanced proficiency in Data Engineering is required.
- Advanced proficiency in Data Access is recommended.
- Advanced proficiency in Data Migrations, Data Pipelines, and Data Processing is suggested.
Licence no: 12C6060