Python/Spark/AI Developer

The opportunity

We're a data collaboration company. Our enterprise clients in financial services and retail use our platform to gain insight from shared data without exposing individual identities. Data and analytics engineering sit at the core of what we deliver.

The role

We're looking for a Python/Spark/AI Developer to replatform legacy T-SQL logic onto Spark and Delta Lake pipelines and to build the APIs and AI features around them. You'll write type-safe, spec-first Python.

What you'll do

• Pipelines: Build Spark and PySpark pipelines on Delta Lake, re-expressing T-SQL logic in Spark SQL.

• Code quality: Write type-hinted, tested Python with pytest suites and CI lint and type-check gates.

• APIs: Build REST APIs with FastAPI for pipeline jobs, including auth, idempotency and job-status semantics.

• Parity: Validate migrated pipelines against the legacy system with parity evidence.

• Specs: Work against design docs and ADRs, and document changes as you go.

What you bring

• Python 3.12 as a primary language, with type-hinted code that passes strict static analysis (pyright or mypy), Pydantic models, and modern packaging (uv or Poetry)

• Apache Spark and PySpark experience building production pipelines on open-source Spark, including the DataFrame API, Spark SQL, partitioning and performance tuning

• Delta Lake or a comparable lakehouse format (Iceberg, Hudi), including MERGE INTO, schema evolution, time travel and idempotent writes

• Strong, dialect-portable SQL, with the ability to read T-SQL and reproduce its semantics faithfully in set-based logic

• Automated testing with pytest, including fixtures, markers and tiered unit, integration and end-to-end suites

• REST API development with FastAPI or equivalent, including request validation and signed-request auth

• Docker-based development against Compose stacks

• Git and CI discipline: PR-driven work with ruff, type-check and test gates

• 4+ years of professional Python experience, with 2+ years building Spark or comparable distributed pipelines in production

• Ability to work autonomously from written requirements and specifications

• Bachelor's degree in Computer Science, Engineering or equivalent experience

Nice to have

• LLM integration: provider-neutral AI features, structured output validation, and local or hosted inference

• Data governance and privacy engineering: PII tokenisation, k-anonymity, GDPR-style data handling

• Lakehouse platform components such as Hive Metastore, Trino or Apache Ranger

• Azure data estate familiarity: ADLS Gen2, Synapse or ADF concepts, Key Vault

• Observability tooling: structured logging, OpenTelemetry, OpenLineage

• Migration or parity-testing experience

• Spark certification, such as Databricks Certified Developer

Logistics

• Location: Remote. Your working hours should overlap with Poland's time zone (CET) within ±2 hours.

• Employment: Full-time

• Compensation: Not disclosed

• Benefits: Wellness programme, home-office reimbursement, continuous learning, and performance incentives

• How to apply: Through TechTree. One application can also match you to other relevant roles.