Data Engineer
Summary
Designs and maintains data pipelines, ETL processes, and dashboards using SQL, Python, Databricks, and visualization tools to ensure data quality and compliance.
Responsibilities
. Responsible for designing, developing, and maintaining data solutions for data generation, collection, and processing.
. Creating data pipelines, ensuring data quality, and implementing ETL processes to migrate and deploy data across systems.
. You are expected to be a subject matter expert, collaborate and manage the team to perform effectively.
. Engage with multiple teams, and contribute to key decisions while providing solutions to problems for your immediate team and across multiple teams.
. Query, clean, and transform datasets using SQL and Python on Databricks / Pyspark .
. Ensure data quality and consistency across systems in accordance with government Instruction Manual (IM8) standards.
. Build, enhance, and maintain dashboards for data monitoring, performance reporting, and user behaviour analytics.
. Partner with business stakeholders to understand data requirements and translate them into impactful visual insights.
. Documentation & Knowledge Management
. Develop clear and comprehensive user guides for dashboards, charts, and data workflows in Databricks
. Maintain technical documentation (e.g., on Confluence) for analytics processes, standards, and best practices.
Requirements . Bachelor's degree in Computer Science, Engineering, or related field. . At least 4-5 years of relevant working experience . Proficiency in SQL and Python for data processing and automation. . Hands-on experience in dashboard tools such as Tableau, Power BI, or equivalent visualisation tools. . Familiarity with Databricks or similar big data/analytics platforms. . Ability to produce high-quality technical documentation. . Experience in system testing, SIT/UAT, or quality assurance processes. . Knowledge of Spark for distributed data processing. . Understanding of Git or other version control systems. . Advanced proficiency in Data Engineering is required. . Advanced proficiency in Data Access is recommended. . Advanced proficiency in Data Migrations, Data Pipelines, and Data Processing is suggested.
Licence no: 12C6060
Requirements . Bachelor's degree in Computer Science, Engineering, or related field. . At least 4-5 years of relevant working experience . Proficiency in SQL and Python for data processing and automation. . Hands-on experience in dashboard tools such as Tableau, Power BI, or equivalent visualisation tools. . Familiarity with Databricks or similar big data/analytics platforms. . Ability to produce high-quality technical documentation. . Experience in system testing, SIT/UAT, or quality assurance processes. . Knowledge of Spark for distributed data processing. . Understanding of Git or other version control systems. . Advanced proficiency in Data Engineering is required. . Advanced proficiency in Data Access is recommended. . Advanced proficiency in Data Migrations, Data Pipelines, and Data Processing is suggested.
Licence no: 12C6060