AI Data Engineer

AI Data Engineer

Location: India(Remote)
Department: Data & AI Engineering
Experience: 5-8+ Years, including 2+ years in production AI/ML or LLM-era data infrastructure
Level: Mid-Senior Individual Contributor
Reporting To: Principal AI Data Architect

About the Role: We are looking for a hands-on AI Data Engineer to build, operate, and optimize production-grade data pipelines, retrieval infrastructure, and ML/LLMOps foundations supporting conversational AI assistants, dashboard copilots, autonomous agents, RAG applications, and predictive ML models. The role involves translating architecture designs and data contracts into reliable, observable, measurable, and continuously improving AI data systems.

Key Responsibilities

• Data Pipeline Engineering
• Build, test, and maintain batch and real-time pipelines using Snowflake, PySpark, Delta Lake, and Kafka.
• Implement data quality checks, schema validation, and pipeline alerting.
• Migrate legacy ETL/data warehouses to AWS/Azure cloud-native architectures, optimizing latency and cost.
• Maintain CI/CD pipelines for automated testing, deployment, rollback, and IaC using Terraform and GitHub Actions.
• RAG, Vector and Retrieval Infrastructure
• Build document ingestion, embedding pipelines, vector store management, and hybrid retrieval using Pinecone, FAISS, ChromaDB, and OpenSearch.
• Optimize chunking, metadata filtering, and re-ranking for precision, recall, and latency.
• Maintain data freshness and index consistency; monitor context relevance and faithfulness.
• Semantic and Knowledge Infrastructure
• Implement business entity mappings, ontologies, and Neo4j knowledge graphs based on Architect designs.
• Build and version feature stores and semantic data contracts for ML models and LLM applications.
• Manage metadata, data lineage, and audit trails.
• ML/LLMOps and Agentic Infrastructure
• Build ML data infrastructure for training data curation, feature engineering, MLflow experiment tracking, and dataset versioning.
• Support LLM fine-tuning through corpus curation, quality filtering, and dataset formatting.
• Automate evaluation for factual accuracy, hallucination detection, and regression; monitor pipeline health and model metrics.
• Build data APIs, tool schemas, memory/state stores, text-to-SQL layers, semantic query interfaces, and context APIs for autonomous agents and conversational AI.
• Implement agent observability by capturing inputs, retrieved context, tool calls, reasoning traces, and outputs.
• Governance, Security and Data Quality
• Implement RBAC, attribute-based access controls, PII detection/masking, data classification, and audit logging.
• Enforce data contracts and schema governance, including breaking-change detection and versioned migrations.
• Monitor data completeness, freshness, and consistency with automated alerting and root-cause analysis.
• Support compliance readiness through audit trails, data provenance, and regulatory documentation.

Must Have Experience

• Proven experience building scalable production batch and streaming data pipelines.
• Strong expertise in Python, SQL, PySpark, Snowflake, Delta Lake, Kafka, and Spark Structured Streaming.
• Experience with AWS/Azure and production RAG infrastructure, including vector stores, embeddings, and retrieval pipelines.
• Working knowledge of MLOps, MLflow, AI CI/CD, automated evaluation, and production monitoring.
• Strong understanding of data governance, quality frameworks, and compliance-aligned engineering.

Technical Skills and Proficiency Levels

• Expert: Python, SQL, PySpark, Kafka, Delta Lake, Snowflake, Docker, Kubernetes, GitHub Actions, AWS (S3, Glue, Kinesis, EKS, Redshift).
• Strong: LangChain, LlamaIndex, LLM APIs (OpenAI, AWS Bedrock, Claude, Hugging Face), Pinecone, FAISS, ChromaDB, OpenSearch, MLflow, FastAPI, Neo4j.
• Solid: CI/CD pipelines, CloudWatch, Grafana, data lineage platforms, MCP (Model Context Protocol).
• Familiar: LangGraph, prompt engineering, RLHF dataset preparation, LLM fine-tuning workflows.

Preferred Qualifications

• B.Tech/M.Tech/M.S. in Computer Science, Data Engineering, or a related field.
• Experience in financial services, payments, cybersecurity, or healthcare, particularly in regulated data environments.
• Hands-on experience evaluating LLM/RAG outputs for faithfulness, relevance, and hallucination rate.
• Exposure to agentic AI infrastructure, agent memory stores, tool API design, LangGraph, AutoGen, and CrewAI.

Expected Mindset

• Deliver tested, observable, documented production code and maintainable pipelines.
• Make downstream AI consumer needs central to engineering decisions.
• Treat data quality as a product requirement and implement measurable monitoring.
• Translate architecture blueprints into reliable implementations.
• Ensure test coverage, observability, and runbooks are completed before considering work finished.

Technology Stack

Delta Lake, PySpark, Kafka, Spark Structured Streaming, Snowflake, AWS (S3, Glue, EKS, Bedrock, Kinesis, Redshift, Lambda), Azure, Kubernetes, Docker, Terraform, GitHub Actions, Jenkins, MLflow, LangChain, LlamaIndex, Hugging Face, OpenAI, Claude, Pinecone, FAISS, ChromaDB, OpenSearch, Neo4j, FastAPI, Python, SQL, MCP, LangGraph, MLOps, CI/CD, Grafana, and CloudWatch.