Senior Data Engineer
Summary
Design and build data pipelines, ETL workflows, and reporting solutions using Python, PySpark, SQL, and Cloudera tools to support analytics and dashboards for a government authority.
Key Responsibilities
- Analyse the Authority’s data needs and document the requirements.
- Refine data collection/consumption by migrating data collection to more efficient channels.
- Plan, design and implement data engineering jobs and reporting solutions to meet the analytical needs.
- Develop test plan and scripts for system testing, support user acceptance testing.
- Build reports and dashboards according to user requirements
- Work with the Authority’s technical teams to ensure smooth deployment and adoption of new solution.
- Ensure the smooth operations and service level of IT solutions.
- Support production issues
- Apply AI coding tools such as Copilot and Coder AI to support development, testing, and code reviews.
- Use automation to improve development efficiency. This includes build, test, and deployment activities.
- Leverage AI assistance to improve code quality. This covers refactoring, defect resolution, and documentation.
- Demonstrate willingness to adopt new AI and automation tools where relevant to work outcomes.
What we are looking for
- Good understanding and completion of projects using waterfall/Agile methodology.
- Working experience using Cloudera AI (Machine Learning), Plotly and data engineering AI tool stack
- Strong SQL, data modelling and data analysis skills are a must.
- Hands-on experience in big data engineering jobs using Python, Pyspark, Linux, and ETL tools like Informatica.
- Hands-on experience in a reporting or visualization tool like SAP BO and Tableau is must.
- Hands-on experience in DevOps deployment and data virtualisation tools like Denodo will be an advantage.
- Hand-on experience in scripting like python and shell scripts will be an advantage
- Track record in implementing systems using Hive, Impala and Cloudera Data Platform will be preferred.
- Good understanding of analytics and data warehouse implementations.
- Ability to troubleshoot complex issues ranging from system resource to application stack traces.
- Track record in implementing systems with high availability, high performance, high security hosted at various data centres or hybrid cloud environments will be an added advantage.
- Passion for automation, standardization, and best practices.
- AI and automation scope past experience will be an advantage.