Data Engineer
Summary
Maintain and secure a big-data platform (Cloudera stack) and automate operations, ensuring high availability and performance for data workflows.
- Oversee day-to-day operations of big data platform to ensure high availability, reliability and performance.
- Proactively monitor big data platform services, components and clusters to identify potential issues. Take corrective actions as needed to maintain platform health
- Manage configuration, upgrades, and patching of big data platform, ensuring all services are up to date
- Work with the Authority's technical teams to ensure smooth deployment and adoption of new solution to support data ingestions, process and workflows
- Maintain clear and detailed documentation of platform configuration, troubleshooting steps and incident resolution.
- Continuously monitor for and address platform security vulnerabilities. Implement patching strategies to resolve identified vulnerabilities and maintain a secure environment.
- Develop automation script to streamline administrative tasks, platform health and ensure operational consistency.
- Ensure the smooth operations and service level of IT solutions.
- Support production issues Requirements
- Hands-on experience, knowledge and troubleshooting of Cloudera Data Platform such as HDFS, YARN, HIVE, Spark, Impala, Ranger, operating systems, security and network.
- Hands on experience with monitoring tools like Cloudera Manager, Zabbix, Grafana, Splunk, SyslogNG
- Familiarity with middleware applications i.e. Informatica, Denodo and scripting languages like bash, python, or shell scripting for automation
- Experience with cloud technology i.e. AWS, Azure is a plus
- Ability to troubleshoot complex issues ranging from system resource to application stack traces.
- Track record in implementing systems with high availability, high performance, high security hosted at various data centres or hybrid cloud environments will be an added advantage.
- Cloudera Certified Administrator or similar certification are a plus.
- Excellent communication skill to work with cross-functional teams
- Ability to handle high-pressure situations and manage critical incidents