Lead Data Engineer

Nasze wymagania:

• Strong hands-on experience with PySpark (DataFrame APIs, tuning, debugging).
• Practical knowledge of Apache Iceberg, including:
• Optimisation techniques (partitioning strategy, file sizing, layout).
• Internal architecture understanding (snapshots, manifests, schema evolution).
• Maintenance operations (rewrite/compaction, orphan file cleanup, metadata handling).
• Solid software engineering fundamentals: clean, testable, maintainable code, problem-solving.
• Experience with CI/CD pipelines, observability, stability improvements in production systems.
• Ability to work in global teams, excellent communication skills.
• Ownership mindset and responsibility for end-to-end delivery.
• Fluency in English (spoken & written).

O projekcie:

We are seeking an experienced Lead Data Engineer to join our client's global development team and drive the design and implementation of next-generation data platforms. You will work closely with teams in Poland, the UK, India, and China to enhance platform capabilities, improve performance, and ensure operational excellence.

Zakres obowiązków:

• Analyse and capture functional & non-functional requirements for near-term deliveries and long-term evolution.
• Design and implement data pipelines and table management workflows using PySpark and Apache Iceberg.
• Drive Apache Iceberg performance and cost optimisation (e.g., file sizing, partitioning, clustering, query efficiency).
• Manage and improve Apache Iceberg maintenance (compaction, rewrite, snapshot/metadata management, orphan file handling).
• Apply strong understanding of Iceberg internals (metadata layout, schema evolution, partition specs).
• Ensure high engineering standards via code reviews, test automation, CI/CD, observability, and production readiness.
• Provide support for owned services and datasets, including on-call duties.
• Mentor and coach junior engineers; lead technical design discussions.
• Note: Detailed project information will be shared during the recruitment process.

Oferujemy:

• Flexible cooperation model
• Hybrid work setup – 6 days a month from the office in Kraków
• Collaborative team culture – work alongside experienced professionals eager to share knowledge.
• Continuous development – access to training platforms and growth opportunities.
• Comprehensive benefits – including Interpolska Health Care, Multisport card, Warta Insurance, and more.
• High quality equipment – laptop and essential software provided.