TATA CONSULTANCY SERVICES ASIA PACIFIC PTE. LTD.

Discussion

Databricks (Delta Lake, PySpark) AI Dev

Position Overview

The Data & AI Engineer role is responsible for building scalable data pipelines, developing AI agents, and delivering end-to-end data and AI applications on the enterprise data platform, with a primary focus on Databricks.

This role bridges data engineering, AI engineering, and data science to deliver production-grade solutions that power analytics, automation, and intelligent applications across Suntory Beverage & Food’s global operations.

As part of the GHQ AAA team, you are expected to contribute to the organisation’s enterprise AI architecture by delivering reusable, scalable, and high-performance data and AI solutions.

Key Responsibilities

Data Pipeline Development

  • Design, develop, and maintain scalable data pipelines using Databricks (Delta Lake, PySpark, workflows).
  • Build robust ETL/ELT processes to ingest, transform, and serve data from multiple enterprise sources (SAP, external data, APIs).
  • Ensure data quality, reliability, and performance optimisation across pipelines.
  • Implement data models aligned with analytics and AI use cases (e.g., feature-ready datasets).
  • Collaborate with data governance teams to ensure compliance with enterprise data standards.

AI Agent Development

  • Design and develop AI agents and GenAI solutions (e.g., knowledge assistants, automation agents) using Databricks and Azure AI capabilities.
  • Implement Retrieval-Augmented Generation (RAG), prompt engineering, and orchestration logic for enterprise AI use cases.
  • Integrate agents with enterprise data sources, vector databases, and APIs.
  • Collaborate with business teams to identify and deliver AI-driven automation and productivity use cases.
  • Ensure scalability, performance, and responsible AI practices in agent deployment.

Databricks App Development

  • Develop end-to-end data and AI applications using Databricks (e.g., notebooks, dashboards, apps, APIs).
  • Build interactive analytics or AI-driven applications for business users (e.g., demand planning tools, decision support apps).
  • Expose data and AI services via APIs (e.g., Databricks Model Serving, API integration).
  • Work with front-end or BI teams (e.g., Power BI, apps) to integrate backend logic into user-facing solutions.
  • Optimise application performance and ensure scalability for enterprise usage.

Data Science Development

  • Support development of machine learning models (forecasting, optimisation, classification).
  • Perform feature engineering and data preparation for modelling.
  • Collaborate with Data Scientists to productionise models within Databricks environment.
  • Integrate trained models into pipelines and applications for real-time or batch inference.
  • Ensure alignment between modelling outputs and business requirements.

Experiences & Skills Requirements

  • Bachelor’s or Master’s degree in Computer Science, Data Engineering, Data Science, or related field.
  • 1–3 years of experience in data engineering, AI engineering, or related roles.
  • Strong hands-on experience with Databricks (Delta Lake, PySpark, workflows, model serving).
  • Proficiency in Python and SQL for data processing and application development.
  • Experience building data pipelines and ETL processes in large-scale environments.
  • Practical experience in AI/GenAI development (LLMs, RAG, prompt engineering).
  • Familiarity with API development and integration for data and AI services.
  • Understanding of machine learning concepts and model deployment workflows.
  • Experience with MLOps / DataOps practices (CI/CD, pipeline automation, monitoring) is preferred.
  • Knowledge of Azure ecosystem (ADLS, ADF, Azure AI, API Management) is a plus.
  • Strong problem-solving skills and ability to work across cross-functional teams.
  • Experience in FMCG, supply chain, or commercial analytics is an advantage.
Show