Senior Data Engineer

We are looking for an experienced Senior Data Engineer to join the Encarta squad within the Personalization mission. You'll drive the deprecation of legacy batch affinity datasets, well-scoped, high-impact work that will save approximately $1.58M/year in GCP costs. You'll migrate downstream consumers to a modern event-driven stack, confirm data parity, and tear down the underlying infrastructure. The Encarta squad owns User Behavior Knowledge (UBK), the platform that powers how users listen to, watch, and care about content, so your work will directly improve the cost efficiency and maintainability of one of the most critical personalization systems. Above all, your work will impact the way the world experiences music.

WHAT YOU'LL DO:

• Audit and migrate downstream consumers of 16+ legacy batch music affinity endpoints (7-day, 28-day, and 6-month user-artist and user-track aggregations) to existing Mirage view exports.
• Deprecate legacy audiobook and podcast affinity and listening history endpoints (~28 endpoints total), replacing them with their new system equivalents and confirming data parity.
• Tear down the Dawnshard system once its upstream dependencies (music and audiobook affinity) are fully deprecated.
• Coordinate with downstream teams to acknowledge and validate migrations, driving the communication and ensuring a smooth transition.
• Decommission legacy BigQuery endpoints and the descriptor-affinity pipeline, eliminating ongoing maintenance burden and incident risk.
• Progressively unlock cost savings throughout the engagement, with the first $60K/month reduction target within the first two months.

WHO YOU ARE:

• You are an experienced software engineer with a strong background in data engineering, comfortable owning end-to-end deprecation and migration work independently.
• You are proficient in Scala and Java and have experience working with data pipelines and batch processing frameworks.
• You are comfortable working with SQL and data analytics platforms such as BigQuery, including auditing endpoint usage, validating data parity, and managing dataset lifecycles.
• You have experience with GCP infrastructure — cloud storage, compute, and cost management are familiar territory for you.
• You have a track record of migrating or deprecating production systems safely: auditing downstream consumers, coordinating stakeholders, and decommissioning infrastructure without disruption.
• You are comfortable navigating a large, complex codebase and can quickly understand existing systems enough to tear them down responsibly.
• You appreciate clear scope and milestones and can work autonomously toward them with minimal hand-holding — this role is scoped as "easy for an embed" across all items.
• Quality and reliability matter to you — you understand the stakes of deprecating systems that serve live personalization data, and you take a methodical, well-documented approach.
• You communicate proactively, especially when coordinating migration timelines with other teams.
• You are comfortable working in an agile environment with practices like continuous delivery, automated testing, and defensive programming.

Start/end: 2026-09-28 to 2027-03-27

Workplace: Remote within Sweden.