Senior Manager, Data Engineering
Summary
Lead a 10+ person data engineering team and own the end-to-end data platform — unified ingestion framework, Apache Iceberg lakehouse, dbt modeling layer, and Trino query engine — while staying hands-on in design and code reviews.
You will own how data gets into our platform and how it gets served back out — ingestion, the lakehouse, and the query layer underneath everything analytics and product depend on.
The problem is specific. Data arrives from CDC streams, transactional databases, event topics, partner APIs, and files, and today each source carries its own pipeline, its own failure modes, and its own on-call story. Your mandate is to collapse that into one ingestion framework and one open lakehouse — reliable enough to publish SLOs against, fast enough to serve interactive query, and cheap enough to defend line by line.
This is a hands-on role. You will set technical direction, hire, and grow the team — and you will also be in design reviews, in code review, and in the pipeline when a stateful stream will not recover from its checkpoint. Expect roughly half your time in technical work.
You own the full path: how data lands, the table format and its lifecycle, the transformation layer, the engines that serve it, and the SLOs on top of all of it. When a dataset is late or wrong, it is your team's phone that rings — and you are expected to have already built the thing that catches it first.
What you will do:
- Lead, hire, and grow a team of 10+ data engineers — set the technical bar through design and code review, not through status meetings.
- Own the architecture and delivery of a unified ingestion framework: one configuration-driven path for batch, CDC, and streaming sources, with schema evolution, replay and backfill, idempotency, dead-letter handling, and data contracts built into the framework rather than reimplemented per pipeline.
- Own production Spark Structured Streaming pipelines — watermarking, stateful joins and aggregations, checkpoint and restart discipline, exactly-once sinks, lag and backpressure management.
- Own the Apache Iceberg lakehouse: partition and sort strategy, file sizing and compaction, snapshot and orphan-file lifecycle, schema and partition evolution, and multi-engine interoperability.
- Set the dbt modeling standard — layering conventions, tests, contracts, CI enforcement, and lineage that stakeholders trust.
- Own Trino catalog design, workload isolation, and query performance for interactive and federated access.
- Define and meet freshness, completeness, and latency SLOs. Run a 24x7 on-call rotation with a short mean time to restore.
- Own cost: a defensible cost-per-pipeline and cost-per-dataset number, and the levers to move it.
- Partner with product, analytics, and architecture to sequence the roadmap, and bring rigor to decisions — collect the data, seek dissent, and run pilots rather than arguing from opinion
Required Qualifications
- Bachelor's degree in Computer Science, Engineering, or equivalent practical experience.
- 8+ years in data engineering, including 3+ years leading engineers as a manager or tech lead — and you are still hands-on in code and design.
- Production experience with Spark Structured Streaming at scale: state store growth, checkpoint recovery, watermark tuning, and late or out-of-order data.
- Deep Apache Spark and PySpark performance work — diagnosing and fixing skew, shuffle pressure, small-file problems, and executor memory failures on real workloads.
- Experience building or substantially owning a reusable ingestion framework serving multiple source types — not a collection of individual pipelines.
- Production experience with an open table format (Apache Iceberg preferred) including schema and partition evolution, compaction strategy, and migration from an existing format.
- Experience with dbt as a team-wide modeling standard, including testing and CI.
- Experience with Trino or Presto operations and query optimization.
- Experience running reliable, high-scale platform systems — 24x7 on-call, availability targets, and fast restoration of service
Preferred Qualifications
- Apache Flink, Kafka or MSK internals, or high-throughput stream-join design.
- CDC tooling in production (Debezium, DMS, GoldenGate, Qlik) and integrating legacy or mainframe sources into modern pipelines.
- Data contracts, catalog, and lineage tooling (DataHub, OpenMetadata, Glue, Unity).
- Iceberg REST catalog implementations and multi-engine interoperability.
- AWS, Kubernetes, and Terraform fluency — you can debug below the framework layer.
- Multi-tenant B2B data platforms with per-tenant cost attribution and isolation.
- Open-source contribution to the projects in this stack.
What Success Looks Like
-
A stable, well-led SRE organization with clear ownership, career paths, and low regrettable attrition among your managers and their teams.
-
Consistent, Datadog-driven observability and SLOs in place across the organization, with measurable reduction in Sev1/Sev2 incidents and mean time to detect/resolve.
-
Modern, standardized infrastructure practices — GitOps delivery via Argo, IaC via Terraform/OpenTofu, and reliable CI/CD via Jenkins — adopted consistently across teams and clouds.
-
A mature, blameless incident-management culture with strong postmortem follow-through.
-
Strong cross-functional trust with engineering, product, and security/compliance stakeholders.
As published by lever
Which location are you applying for?, Resume/CV, Full name, Email, Phone, Current location, Current company, LinkedIn URL, Twitter URL, GitHub URL, Portfolio URL, Other website
- Are you currently authorized to work in the United States for any employer? choose one
- If yes, do you now or will you in the future require sponsorship for employment visa status (e.g., H-1B visa, transfer, etc.) to work with our company? choose one · optional
- Have you previously worked for PDI Technologies or any company that has since been acquired by PDI? choose one · optional
- Are you currently employed by PDI Technologies or any company that has since been acquired by PDI? choose one · optional
- Were you referred to this position by a current PDI Technologies employee? choose one
- If yes, please provide their name. written answer · optional
- What is your target total compensation (OTE)? Please include: Your desired base salary range (annual, in USD — e.g., $XX,000–$XXX,000), and A range for any bonus or commission you'd like us to consider as part of your overall compensation (e.g., $XX,000–$XX,000 annually). written answer
- How many years of professional data engineering experience do you have? choose one · optional
- How many years of experience do you have formally managing or technically leading data engineers? choose one · optional
- This is a player-coach leadership role, with approximately 50% of your time spent on hands-on technical work, including architecture, design reviews, code reviews, troubleshooting, and production incidents. Please briefly describe how hands-on you are in your current or most recent role. Please answer in 1–3 sentences. written answer
- Have you personally supported Spark Structured Streaming pipelines in a production environment? choose one · optional
- Briefly describe the most complex production streaming pipeline you have owned or supported. Please include the scale and your experience with areas such as checkpoint recovery, watermarking, stateful processing, late or out-of-order data, lag, or backpressure. Please answer in 1–3 sentences. written answer
- Have you built or substantially owned a reusable ingestion framework that supported multiple data sources or ingestion patterns, rather than developing individual pipelines for each source? choose one · optional
- Briefly describe a reusable ingestion framework you have built or substantially owned. What types of sources did it support, such as batch, CDC, streaming, APIs, databases, or files, and what was your specific contribution? Please answer in 1–3 sentences. written answer
- Which of the following technologies have you used hands-on in a production environment? Select all that apply. choose one
- Please describe your production experience with Apache Iceberg or another open table format. Include any experience with schema or partition evolution, compaction, file sizing, snapshot management, or migration from another storage format. Please answer in 1–3 sentences. written answer
- Tell us about a significant production data incident or performance issue you personally helped diagnose and resolve. What was the issue, and what did you do to restore or improve the system? Please answer in 1–3 sentences. written answer
- In your current or most recent role, approximately what percentage of your time is spent on hands-on technical work versus people leadership? Please provide an example of technical work you have personally completed within the last 12 months. Please answer in 1–3 sentences. written answer