Data engineer platformy CDP

Описание:

For our client in the fuel and energy sector, we are looking for a Data Engineer to join the team responsible for developing and maintaining a CDP platform based on a Data Lakehouse architecture in Google Cloud Platform.

Задачи:

• Supporting the project and maintenance team in building and developing a CDP platform using a Data Lakehouse architecture (GCP)
• Designing, implementing and optimizing ETL/ELT pipelines (batch and streaming)
• Creating and managing data transformations across the Bronze, Silver and Gold layers (medallion architecture)
• Orchestrating data flows and monitoring jobs
• Working with CDC mechanisms and replicating data from transactional sources to BigQuery
• Ensuring data quality, cataloguing metadata and complying with Data Governance principles, including GDPR
• Working closely with IT teams and technical Project Managers in an international environment

Требования:

• Experience in at least several cloud solution implementation projects in a Data Engineer role
• At least 6 months of practical experience working with GCP (Google Cloud Platform) technologies
• Very good knowledge of key GCP Data & Analytics services: BigQuery (partitioning, clustering, external tables, query optimization), Cloud Storage, Cloud Dataflow (Apache Beam) and Cloud Composer (Apache Airflow)
• Knowledge of services supporting data integration and management: Datastream (CDC/replication), Dataplex (metadata, Data Quality, Data Lineage), Pub/Sub (streaming) and Dataform (SQL transformations in BigQuery)
• Knowledge of SQL (BigQuery dialect) and Python, including Pandas, for data transformation and analysis
• Experience implementing and maintaining medallion architecture (Bronze/Silver/Gold), ETL/ELT processes and Data Governance principles, including data deduplication and versioning
• Knowledge of Git and CI/CD processes (e.g. Cloud Build), as well as Cloud Monitoring/Logging configuration
• Very good communication and organizational skills, and knowledge of English sufficient to work in an international environment

Условия:

• Work at a renowned organization (ITwiz BEST100) operating according to international quality and security standards (ISO 9001, ISO 27001)
• A non-corporate atmosphere and a flat organizational structure
• The opportunity for continuous development through: Working with a variety of technologies on projects for renowned clients
• Collaborating with an experienced team of experts who are happy to share their knowledge
• Funding for training and certifications
• Employee Referral Program
• An organizational culture based on the following values: We act with good intentions We share knowledge We take responsibility We continuously improve People are our top priority We act with an understanding of the purpose and meaning of our work We communicate honestly and professionally
• Taking part in charitable initiatives, including working with the Joanna Radziwiłł "Caring Wings" Foundation and participating in the Christmas "Gwiazdorzymy" campaign
• A diverse and inclusive work environment - representatives of four generations work together at our company
• Equal development opportunities - women hold 50% of managerial and leadership positions