Associate Director Data Engineering

Associate Director, Data Engineering
Location: Noida, India
Work Mode: 5 days WFO

About CloudKeeper
CloudKeeper is a cloud cost optimization partner that combines the power of group buying &
commitments management, expert cloud consulting & support, and an enhanced visibility &
analytics platform to reduce cloud cost & help businesses maximize the value from AWS,
Microsoft Azure, & Google Cloud.
A certified AWS Premier Partner, Azure Technology Consulting Partner, Google Cloud Partner,
and FinOps Foundation Premier Member, CloudKeeper has helped 350+ global companies
save an average of 20% on their cloud bills, modernize their cloud set-up and maximize value.
CloudKeeper Platform Suite is a leading enterprise FinOps platform that helps organizations
optimize and govern their multi-cloud spend across AWS, Azure, and Google Cloud. Our
platform processes billions of dollars in cloud spend, delivering actionable cost optimization
recommendations, automated governance, and financial accountability to enterprises
worldwide.
To know more - https://www.cloudkeeper.com/platform-suite

The Opportunity
CloudKeeper's products are federated, but they stand on one shared foundation: cost and
usage data. Every CK-Lens dashboard, every CK-Commit purchase decision, every CK-Tuner
recommendation, and every FinOps for AI insight is only as good as the billing data underneath
it. Today that data flows through pipelines that grew up inside individual products, spanning the
Stormus backend, Jenkins-driven data jobs, and Airflow DAGs.
We are hiring an Associate Director, Data Engineering to own this layer as a product in its own
right. You will define the target architecture for a multi-cloud billing data platform, consolidate
ingestion across AWS, Azure, and Google Cloud, set the warehouse and lakehouse strategy,
and build the team that runs it. The bar is unforgiving: customers make financial decisions on
these numbers, so every figure has to reconcile to the invoice.

What You'll Own
Multi-Cloud Billing Data Platform
- A single ingestion layer for AWS Cost and Usage Reports and Data Exports, Azure Cost
Management exports and Billing APIs, and GCP BigQuery billing exports, on daily and
intra-day schedules
- Normalization of provider-specific line items into a common cost model, aligned to the
FinOps Foundation's FOCUS specification
- Correct handling of restatements, late-arriving usage, credits, discounts, marketplace
charges, and consolidated billing across payer accounts, so reported cost always
matches what the customer is invoiced
Warehouse and Lakehouse Architecture
- The analytics engine behind 100+ CK-Lens dashboards, anomaly detection, custom
reports, and LensGPT natural-language queries, which runs on Snowflake today
- The target architecture for the next three years: where a warehouse ends and a
lakehouse begins, which workloads belong in each, and how the platform scales with
customer count without scaling spend at the same rate
- Warehouse and lake cost governance. CloudKeeper sells data-platform cost
optimization, and the platform should be the proof
Data Foundation for New Products
- Ingestion for FinOps for AI: GPU instance usage, managed AI services such as Bedrock,
Azure OpenAI, and Vertex AI, and LLM API consumption from providers such as OpenAI
and Anthropic
- Kubernetes cost telemetry from Prometheus and OpenCost feeding CK-Tuner
Kubernetes
- Cost allocation and virtual-tagging datasets that power unit economics reporting
- A governed semantic layer and data contracts so that LensGPT and the LensGPT MCP
server can answer questions about cost data reliably

Responsibilities
Data Engineering Leadership
- Build and lead the data engineering pod: hire and mentor Data Engineers and Senior
Engineers in data engineering, set coding, testing, and review standards, and grow the
team as the product lines scale

- Own delivery against a data-platform roadmap agreed with Product Management, with
explicit data contracts for each product and published service levels for data freshness
and accuracy
- Run the platform like a product: define and report its metrics, including data freshness,
reconciliation accuracy against source billing, pipeline reliability, and warehouse cost per
customer, and review them monthly
- Represent data engineering in architecture, security, and customer conversations,
including due-diligence and security questionnaires from enterprise customers
Technical Strategy
- Define the target data architecture across batch and streaming, warehouse and lake,
and three cloud providers, and sequence the migration from today's product-embedded
pipelines without disrupting delivery
- Design the reconciliation and data-quality framework: every pipeline proves its output
against the source invoice, and every restatement is traceable
- Establish data governance that satisfies CloudKeeper's SOC 2 Type 2 controls and UK
GDPR obligations: access control, retention, lineage, and customer data isolation
- Build the integration points with CK-Lens for visibility, CK-Commit and CK-Tuner for
optimization, and the FinOps for AI and Kubernetes products for new data domains
- Lead the team's adoption of AI-assisted development. CloudKeeper builds AI-first and
expects measurable output gains from AI in the software development lifecycle

Technical Landscape You'll Navigate
Cloud Billing and Usage Sources
- AWS: Cost and Usage Reports, Data Exports, Organizations consolidated billing, S3,
CloudWatch
- Azure: Cost Management exports, Billing and Consumption APIs, Azure Monitor
- GCP: BigQuery billing export, Cloud Billing APIs
- Standards: FinOps Foundation FOCUS specification
Warehouse and Lakehouse
- Warehouse: Snowflake (current analytics platform); BigQuery, Redshift, or Synapse
experience relevant
- Lake and table formats: S3, ADLS, or GCS object storage with Apache Iceberg, Delta
Lake, or Hudi; Parquet
- Modeling and transformation: dimensional and semantic-layer modeling; dbt or
equivalent transformation frameworks

Processing and Orchestration
- Compute: Apache Spark (SQL, DataFrames, Structured Streaming); Hadoop
ecosystem familiarity
- Orchestration: Apache Airflow; Jenkins-driven jobs in the current estate
- Messaging: Apache Kafka, RabbitMQ, AWS SQS
- Languages: Python plus Java or Scala
Operational Stack
- Application data: MySQL, MongoDB, behind Java and Spring Boot services
- Observability: Prometheus, Grafana, OpenTelemetry, OpenCost
- Platform: Kubernetes, multi-tenant SaaS on AWS with Azure and GCP footprints

Role Requirements
Experience
- 10+ years in data engineering with 3+ years leading data engineering teams of 410
people building and shipping SaaS or data products. Leadership experience in services,
consulting, or systems integration (SI) teams does not apply
- Hands-on production experience with the data ecosystems of at least two of AWS,
Azure, and Google Cloud
- Has built and operated both a cloud data warehouse (Snowflake, BigQuery, Redshift, or
Synapse) and a data lake or lakehouse (Iceberg, Delta Lake, or Hudi on object storage)
at multi-terabyte scale, and can explain when to use which
- Track record of pipelines whose output reconciles to a financial source of truth, such as
billing, payments, or ledgers, strongly preferred
- Experience with cloud billing data (Cost and Usage Reports, Azure cost exports,
BigQuery billing export) or FinOps practice is a significant advantage
Technical Depth
- Expert-level Spark and SQL; strong Python, with Java or Scala
- Warehouse performance and cost engineering: clustering, partitioning, materialization
strategy, query cost governance
- Streaming with Kafka and batch orchestration with Airflow in production
- Data modeling for analytics: dimensional models, semantic layers, and data contracts
consumed by multiple products
- Data quality, lineage, metadata, and governance tooling, and the judgment to apply them
without slowing delivery
- Working understanding of SOC 2 and GDPR obligations as they apply to a data platform

Leadership
- Ability to operate in a "founding team" mode: small team, high ownership, rapid shipping
- Experience hiring and building data engineering teams from scratch
- Comfortable wearing multiple hats: architecture, coding, hiring, product input
- Strong communicator who can influence product, platform, security, and customer-facing
teams, and can explain a reconciliation gap to a CFO as clearly as to an engineer
- Sets the expectation for AI-assisted development by using it personally

Why CloudKeeper
- Market Opportunity: Cloud spend optimization is a $10B+ market growing rapidly as
enterprises accelerate cloud adoption
- Data at the Core: The platform processes billions of dollars in cloud spend. Data
engineering is the foundation every product stands on, not a support function
- Platform Ownership: Define and build the shared data platform for a federated product
portfolio within a revenue-generating company: startup speed with enterprise stability
- New Domains: FinOps for AI, Kubernetes cost, and data-platform cost optimization are
new data problems being solved for the first time, and this role owns their data
foundation
- Impact: The north-star metric for the whole product line is savings delivered per
customer per month. Every improvement in data accuracy, freshness, and coverage
moves it directly
- Growth: Build the team and the platform from their current state to scale, with the
metrics and ownership that come with running it as a product

CloudKeeper is an equal opportunity employer. We celebrate diversity and are committed to
creating an inclusive environment for all employees.