Senior Big Data Engineer
Summary
Senior Big Data Engineer builds and maintains real-time data pipelines and ETL processes using Python/Java, Spark, Kafka, and cloud/distributed systems to process and analyze large datasets for a gaming company.
At GiG we are looking to hire a Senior Big Data Engineer for a 6-month project in our Data department. You will assume a pivotal role in taking ownership of the analysis, design, building, support of major realtime big data projects and contribute to
the team's success through code reviews, promoting best practices and
implementing robust solutions across all aspects of big data endeavours.
Embracing this vital role, the Senior Big Data Engineer is instrumental in driving data-driven excellence, harnessing the power of information for strategic decision-making, and fostering organisational growth and success.
the team's success through code reviews, promoting best practices and
implementing robust solutions across all aspects of big data endeavours.
Embracing this vital role, the Senior Big Data Engineer is instrumental in driving data-driven excellence, harnessing the power of information for strategic decision-making, and fostering organisational growth and success.
- Design, create, and maintain ETL processes, data integrations, and data pipelines
- Ensure seamless handling of vast amounts of data.
- Leverage extensive knowledge, the engineer develops high-performance services to efficiently process, store, and analyse large datasets
using optimised code. - Aim to maintain data accuracy and reliability, takes the lead in designing and implementing data validation and quality checks tailored to meet specific business needs.
- Troubleshoot live production systems is a regular part of the engineer's responsibilities, ensuring smooth operations and swift issue resolution.
- Demonstrate an innovative spirit, the engineer actively tests and
implements new technologies, staying abreast of the latest advancements relevant to current and future projects.
- Strong object oriented programming skills - ideally Python and/or Java
- Experience with big data technologies such as Spark, Kafka, Nifi, and Big
- Data datastores
- Strong understanding of schema design methodologies and data models
- Expert in extracting large datasets using SQL, familiarity with SQL
- databases
- Strong experience with cloud computed and/or distributed systems