data engineer in real-time big data
Posted 4
views
Summary
Data engineer in GiG's Data department in Barcelona, building and maintaining real-time big data ETL pipelines, integrations, and high-performance services to process and analyse large datasets, plus troubleshooting live production systems. Core stack: Spark, Kafka, Nifi, ClickHouse, SQL, and Python/Java.
Описание:
GiG’s Data department develops and supports real-time big data projects, using information to support strategic decision-making and organisational growth.
Задачи:
- Design, create, and maintain ETL processes, data integrations, and data pipelines
- Ensure seamless handling of large volumes of data
- Develop high-performance services to process, store, and analyse large datasets using optimised code
- Design and implement data validation and quality checks tailored to business needs
- Troubleshoot live production systems and resolve issues
- Test and implement new technologies relevant to current and future projects
- Review code, promote best practices, and implement robust solutions across big data projects
Требования:
- Strong expertise in big data technologies such as Spark, Kafka, Nifi, ClickHouse, and big data datastores
- Strong understanding of schema design methodologies and data models
- Expertise in extracting large datasets using SQL and familiarity with SQL databases
- Object-oriented programming skills, ideally in Python and/or Java
- Strong experience with cloud computing and/or distributed systems
Условия:
- Great career development opportunities
- International health insurance
- Health and Wellbeing Package (350 EUR per year)
- Birthday day off
- Me Time: 1 day off per year