Senior Software Engineer (Data Platform Engineer, Online Analytical Processing)
Summary
Build and scale Adyen’s on-premise big-data platforms (500+ nodes, 23+ PB storage) using Java, Python, Spark, Kafka, and Druid to power real-time analytics and data pipelines for 10s of products.
- Sitting at the intersection of Data Engineering, Backend Engineering, and Systems Engineering, Data Platform Engineers at Adyen build the foundational layer of tooling and processes for our on-premise Big Data Platforms.
- These tools support 10s of products, 100s of developers, and 1000s of daily jobs that add to Adyen’s strong portfolio of capabilities
- You’ll collaborate with data engineers and ML scientists and engineers to build and roll-out these tools.
- Along with DataOps Engineers, you’ll develop and operate multiple big data platforms, of which the biggest is 500+ nodes, with 25k+ CPU cores, 240+ TBs of RAM, and 23+ PBs of storage capacity, running tens of thousands of spark jobs a day
- We are currently expanding our platform capabilities. Depending on your expertise, your immediate focus will be driving one of these two key ecosystems:
- The OLAP Ecosystem: Scaling and optimizing our real-time analytical powerhouses (Druid, FastAPI, Java, Kafka)
- The Data Engineering Ecosystem: Designing and building data contracts, validation frameworks, optimization tooling, and robust Kafka ingestion pipelines
- You will balance building innovative new tools from scratch with relentlessly improving our existing infrastructure
- Scale: Developing and maintaining high-performance, distributed big data platforms
- Distributed Systems: Working deeply with various flavors of databases, filesystems, and compute engines
- Data Quality & Discoverability: Building tools that enhance metadata collection, data lineage, and platform-wide data quality
- CI/CD & Infrastructure: Optimizing continuous release/deployment environments within a large-scale private cloud system
- Streaming: Developing stream processing applications and frameworks to reduce time‑to‑insight for analysts and data scientists
Benefits
- Global exchange program
- Weekly happy hour
- Delicious healthy lunches
- Phantom share package
- Yearly trip to Amsterdam
- Paid holidays
- Work from home opportunities
- Python or Java Expert: 6+ years of experience with object‑oriented programming
- Distributed compute/data systems (eg, Spark, Trino, Druid, ClickHouse)
- Collaborative Enabler: A strong communicator who thrives on building bridges between the teams you enable (analysts, data scientists) and the teams you depend on (infrastructure, security)
- Real‑time and batch pipelines (Kafka, Spark Streaming) with a focus on user‑friendliness and reliability
- Distributed Systems Architect: Proven experience developing and maintaining: