Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Papigen

Senior Databricks & Governance Engineer

Posted Updated

Description

Papigen is seeking a Senior Databricks & Governance Engineer to build scalable data pipelines, enforce governance, and prepare unstructured data for Generative AI in a modern Lakehouse. You will implement end-to-end ETL/ELT processes, manage data quality, and optimize metadata and security using Unity Catalog and Collibra integrations.

Ideal candidates have 5+ years in data engineering, hands-on Azure Databricks and Delta Lake skills, and experience with GenAI data preparation, embeddings, and

Develop and maintain batch and streaming ETL/ELT pipelines using Databricks, PySpark, Python, SQL, and Delta Lake. Orchestrate data workflows using Azure Data Factory (ADF). Extract and process unstructured data from PDFs, images, logs, and text files. Prepare data for RAG, LLM applications, embeddings, and vector search. Implement data access controls, security, and lineage using Unity Catalog. Integrate Databricks metadata with Collibra using REST APIs. Design and maintain Bronze, Silver, and Gold layers using Medallion Architecture. Develop dimensional and Lakehouse data models. Optimize performance using Z-Ordering, Liquid Clustering, and Delta Lake optimization techniques. Ensure data quality, governance, metadata consistency, and pipeline reliability. 5+ years of Data Engineering experience. Hands-on Azure Databricks expertise. Experience with unstructured data and GenAI preparation. Experience with Delta Lake and Medallion architecture.

Skills

Apply

See also