Data Platform Engineer
Summary
Hands-on Data Platform Engineer on Practo's AI Labs team in Bengaluru, architecting and scaling end-to-end data infrastructure: CDC/ETL-ELT pipelines, Apache Iceberg lakehouse, multi-region data systems, and AI/ML workflows like LLM inference and PII/PHI redaction, built on AWS with Terraform/CDK, Temporal, dbt, SQL and Python.
About the Role:
We are looking for a highly hands-on Data Platform Engineer to join our AI Labs team in Bengaluru. This role will be responsible for architecting, building, and scaling our end-to-end data platform, bridging traditional data engineering with modern AI infrastructure.
You will own the design and implementation of data pipelines, lakehouse architecture, multi-region data systems, and AI/ML-powered data processing workflows. From Change Data Capture (CDC) and ETL/ELT pipelines to LLM inference and multimodal data redaction, you will build reliable, scalable, and cost-efficient data infrastructure.
Roles & Responsibilities:
1. End-to-End Data Architecture & Engineering:
- Design, build, and maintain end-to-end data pipelines, covering CDC ingestion from production databases to production-grade lakehouse storage.
- Develop and scale workflow orchestration using Temporal and data transformation pipelines using dbt.
- Build standardized dimensional data models and optimized relational/materialized views to support analytics and business intelligence.
- Modernize legacy data workflows by replacing traditional DAG runners and cron-based batch jobs with scalable orchestration frameworks.
2. Multi-Region Architecture & Data Residency:
- Design and implement multi-region data architectures across cloud environments, ensuring compliance with data residency and sovereignty requirements.
- Establish secure data isolation mechanisms to ensure sensitive patient and transaction data remains within permitted geographical boundaries.
- Build compliant cross-region replication, synchronization, failover, and data-routing mechanisms.
3. AI/ML Integration & Multimodal Data Processing:
- Integrate machine learning models into data pipelines to identify and redact sensitive PII/PHI across text, images, audio, and video.
- Build and operationalize embedding generation, LLM inference, and semantic search pipelines.
- Develop scalable workflows for AI-powered data processing and multimodal content analysis.
4. Lakehouse Infrastructure & Platform Enablement:
- Design and optimize open lakehouse architectures using Apache Iceberg.
- Implement efficient partitioning, ACID transactions, and time-travel capabilities for large-scale datasets.
- Provision and manage secure, scalable Jupyter Notebook environments for data science and exploratory analytics.
5. Cloud Infrastructure, DevOps & FinOps:
- Design, deploy, and manage AWS infrastructure using Infrastructure as Code (Terraform or AWS CDK).
- Establish cost-monitoring frameworks to track cost-per-outcome across data and AI pipelines.
- Own production reliability through monitoring, alerting, data quality checks, and pipeline freshness tracking.
Required Skills & Technical Expertise:
- Strong experience in data engineering, data platform development, and large-scale distributed data systems.
- Proficiency in SQL and Python for data engineering and pipeline development.
- Hands-on experience building CDC-based ingestion and ETL/ELT pipelines.
- Experience with workflow orchestration frameworks such as Temporal.
- Working knowledge of dbt for data transformation and modeling.
- Experience with Apache Iceberg or other modern lakehouse technologies.
- Experience with AWS cloud infrastructure and Infrastructure as Code tools such as Terraform or AWS CDK.
- Exposure to ML/LLM inference pipelines, embedding generation, and AI-powered data processing.
Hiring Note:
We are hiring for two complementary SE II positions from the same candidate pool. Both positions are individual contributor roles focused on deep technical architecture and execution, with no formal people-management or team-mentoring responsibilities.