Staff Engineer, Data Platform & Infrastructure
Summary
Staff data engineer who designs and operates large-scale batch and streaming data pipelines (Spark, Kafka, Flink, Airflow, Kubernetes) that turn billions of daily ad-auction events into clean, queryable datasets powering reporting, billing, and ML at Index Exchange, a programmatic advertising supply-side platform.
- Process billions of events per day across distributed pipelines
- Power core data platform supporting consumers as (reporting, billing, MLO teams)
- Operate across batch (Spark) and streaming (Kafka / Flink) architectures
- Require careful balancing of:
- data correctness
- processing efficiency
- latency vs cost trade-offs
- Designing pipelines that scale without exploding compute costs
- Managing data correctness at scale (deduplication, late data, joins)
- Building systems that support both:
- historical backfills
- near real-time updates
- Evolving pipelines from centralized processing (Hadoop) toward more distributed and efficient patterns
- Streaming pipelines and Streaming DWs.
- Strong experience in data engineering at scale
- Deep expertise in:
- Distributed Data Processing frameworks
- SQL and data modeling
- Leading Projects end to end
- Experience with:
- Airflow or workflow orchestration
- Kafka or streaming systems
- Managing kubernetes
- Strong understanding of:
- distributed data processing
- data modeling (large-scale datasets)
- performance optimization
- Ability to:
- own pipelines end-to-end
- debug complex data issues
- work in high-scale, evolving environments
- Define data processing standards and patterns across teams
- Lead large-scale pipeline and platform initiatives
- Influence data architecture and modeling decisions
- Drive improvements across:
- reliability
- cost efficiency
- scalability
- Design and operate pipelines using:
- Spark, Flink etc.
- Kafka / Flink (streaming)
- Transform raw event data into:
- cleaned datasets (silver layer)
- business-ready datasets (gold / reporting tables)
- Build and maintain canonical datasets (aggregated datasets, reporting tables)
- Define data contracts and ensure consistency across pipelines
- Support evolving use cases:
- reporting
- billing
- ML / experimentation
- Build and maintain Airflow DAGs for:
- pipeline scheduling
- dependency management
- backfills
- Improve reliability and observability of workflows
- Optimize pipelines for:
- performance (runtime, throughput)
- cost (compute efficiency)
- scalability (data growth)
- Improve:
- partitioning strategies
- data layout
- job execution patterns
- Build pipelines that support:
- incremental updates
- streaming transformations
- aggregation at scale
- Contribute to evolving patterns such as:
- edge aggregation
- streaming → batch convergence
- real-time data availability
- Define and evolve data processing patterns:
- batch vs streaming
- aggregation strategies
- incremental vs full recompute
- Work across:
- Spark (core processing)
- Kafka (transport)
- Flink (streaming compute)
- storage systems (Hadoop / Ceph)
- Kubernetes
- Contribute to:
- data platform architecture decisions
- pipeline standardization
- reusable data processing frameworks
- Influence trade-offs:
- latency vs cost
- correctness vs performance
- compute vs storage
- Data Products
- ML and experimentation teams
- Infrastructure and on-prem systems
- Bachelor/ Master’s Degree in Computer Science or Engineering related fields
- 8+ years of experience as a Software Engineer in enterprise grade, large scale distributed software product development
- 3+ years of work experience designing and building high performance data pipelines and applications using Hadoop/Ceph, Spark/Flink, Hive, Presto/Trino, Kafka, StarRocks/Vertica, Airflow, Kubernetes or other similar technologies
- Deep understanding of design principles of large scale distributed systems and familiar with mainstream big data related technologies and distributed frameworks
- Strong leadership, mentorship, and communication skills, with experience collaborating in a globally distributed, culturally diverse team
- Knowledge of data modeling, data warehousing, streaming data processing, and business intelligence reporting tools
- Experience working in Agile methodologies with continuous integration and delivery as CI/CD
- Experience working with containerization, and virtualization tools such as Kubernetes and Docker
- Comprehensive health, dental, and vision plans for you and your dependents
- Paid time off, health days, and personal obligation days plus flexible work schedules
- Competitive retirement matching plans
- Equity packages
- Generous parental leave available to birthing, non-birthing, and adoptive parents
- Annual well-being allowance plus fitness discounts and group wellness activities
- Commuter benefits and discounts, where available
- Employee assistance program
- Mental health first aid program that provides an in-the-moment point of contact and reassurance
- One day of volunteer time off per year and a donation-matching program
- Bi-weekly town halls and regular community-led team events
- Multiple resources and programming to support continuous learning
- A workplace that supports a diverse, equitable, and inclusive environment – learn more here
Skills
- Accessibility
- Agile
- AI
- Airflow
- Big Data
- Business Intelligence
- Ceph
- CI/CD
- Cloud
- Containerization
- Data Engineering
- Data Modeling
- Data Pipelines
- Data Warehousing
- Distributed Systems
- Docker
- First Aid
- Flink
- Hadoop
- Hive
- Kafka
- Kubernetes
- Machine Learning
- Observability
- Spark
- SQL
- Trino
- Virtualization
- Workflow Orchestration
