Senior AI Data Ingestion Architect (Databricks/Azure)
Posted Updated
Description
EPAM Systems Inc is seeking a Senior AI Solution Engineer in Turkey to design and build automated data ingestion pipelines on Azure Databricks, leveraging MS Foundry and MCP expertise. You will gather structured and unstructured data, automate workflows, and expose clean data models to enterprise AI layers.
Responsibilities include crafting high-throughput ingestion, implementing schema enforcement, and deploying pipelines via Azure DevOps and DABs to support AI agents across AI Search and GenAI
Build high-throughput data ingestion pipelines and data scraping engines using Python, PySpark, and Databricks Auto Loader to collect unstructured document content, external registry feeds, and enterprise data. Structure and expose ingested data within the Databricks semantic layer, organize datasets using Open Knowledge Format standards for seamless consumption by AI Search, Databricks Genie Spaces, and LLM reasoning engines. Automate ingestion pipeline deployments, scraping job schedules, and code releases using Azure DevOps and Databricks Asset Bundles (DABs). Scrape and ingest healthcare and insurance data sources, including regulatory databases, public registries, and unstructured medical/claims files, to support AI agent workflows. Implement schema enforcement, automated ingestion failure alerts, and job telemetry within Databricks Workflows and Azure DevOps. 5+ years building high-volume data ingestion pipelines using Azure Databricks, PySpark and Auto Loader. Hands-on experience developing data scraping routines for unstructured data sources. Experience structuring ingested datasets using Open Knowledge Format within Databricks semantic layers.Skills
Sign in to read and join the discussion.