Senior Data Engineer with Python & AI Level

Hiring: Senior Data Engineer Python & Agentic AI

Positions: Manager / Senior Manager
Experience: 8-13 Years
Locations: Noida
Work Mode: As per business requirements

Job Description

We are looking for experienced Senior Data Engineers with strong expertise in Python, PySpark, Databricks, Cloud Data Engineering and Agentic AI development. The ideal candidate should have hands-on experience designing and developing scalable data engineering solutions, building enterprise-grade data pipelines, and integrating Generative AI and autonomous agent-based workflows into data platforms.

The candidate should be capable of driving technical solutions, developing modern data architectures and leveraging AI-assisted coding tools to improve engineering productivity.

Key Responsibilities

• Design, develop and maintain scalable data pipelines using Python, PySpark and Apache Spark.
• Build and optimize ETL/ELT workflows using Databricks and cloud-based data engineering platforms.
• Design and implement data lakehouse, data warehouse and modern data architectures.
• Develop and optimize complex SQL queries, data transformations and data models.
• Build and deploy Agentic AI applications using frameworks such as LangGraph, LangChain, CrewAI, AutoGen or Semantic Kernel.
• Develop multi-agent workflows with tool calling, API integrations, planning, memory and orchestration capabilities.
• Integrate LLMs, RAG pipelines, vector databases and enterprise data sources into data engineering solutions.
• Leverage AI-assisted development tools such as GitHub Copilot, Cursor and Claude Code to accelerate coding, debugging and development.
• Implement data quality, governance, monitoring and performance optimization practices.
• Work with cross-functional teams and stakeholders to deliver scalable enterprise solutions.
• Follow engineering best practices using Git, CI/CD and collaborative development workflows.

Mandatory Technical Skills

• Strong programming experience in Python.
• Advanced SQL and query optimization.
• Hands-on experience with PySpark / Apache Spark.
• Strong experience in Databricks.
• Cloud Data Engineering experience in Azure / AWS / GCP.
• Practical experience in Agentic AI development and multi-agent workflows.
• Experience with ETL/ELT pipelines and data lakehouse architecture.
• Knowledge of data modelling and data warehousing.
• Experience using AI coding assistants such as GitHub Copilot, Cursor, Claude Code or equivalent.
• Git and collaborative software development practices.

Good to Have

• LangGraph, LangChain, CrewAI, AutoGen or Semantic Kernel.
• Generative AI, LLMs, RAG and prompt engineering.
• Vector databases such as Pinecone, FAISS, Azure AI Search or ChromaDB.
• Kafka, Hadoop and distributed data processing.
• Azure Data Factory, AWS Glue, GCP Dataflow or equivalent.
• Databricks Delta Lake, Unity Catalog and Medallion Architecture.
• Docker, Kubernetes, MLflow and CI/CD.
• Data governance, observability and data quality frameworks.

Experience & Leadership Expectations

Manager: 8-12 years of overall experience with strong hands-on expertise in Python, PySpark, Databricks and Agentic AI.

Senior Manager: 10-13 years of overall experience with strong technical architecture, solution design, delivery ownership and team leadership experience, along with hands-on knowledge of the mandatory technologies.

Candidate Profile

We are looking for professionals who have worked on real-time enterprise data engineering and Agentic AI implementations. Candidates should have practical development experience rather than only theoretical knowledge or certifications.