Data Engineering Manager
Summary
Manages and builds the EisnerAI lakehouse on Azure Databricks and ADLS Gen2 with a medallion architecture — batch/streaming ETL/ELT pipelines, RAG-ready data products via Azure AI Search, governance with Unity Catalog and Purview, plus CI/CD and data quality. Core stack: Python, SQL, PySpark, Delta Lake, Azure Data Factory.
Roles and Responsibilities:
- Build and evolve the EisnerAI lakehouse on ADLS Gen2 and Azure Databricks using a medallion architecture.
- Develop batch and streaming ETL/ELT pipelines for firm source systems, client data, and third-party feeds.
- Create reusable, configuration-driven ingestion frameworks to onboard new data sources rapidly.
- Curate vector-ready data products to ground Retrieval-Augmented Generation (RAG) models via Azure AI Search.
- Implement end-to-end data governance, access controls, and lineage via Unity Catalog and Microsoft Purview.
- Optimize platform performance and manage cloud compute costs through partitioning and Z-ordering.
- Establish CI/CD pipelines, automated data quality checks, and contract testing for pipeline reliability.
Preferred Candidate Profile:
- Experience: Minimum 4 years of hands-on experience building production data pipelines and platforms.
- Education: Bachelor's degree in Computer Science, Engineering, Data Science, or a related technical field.
- Technical Expertise:
- Strong programming skills in Python, SQL, and PySpark at scale.
- Proven delivery on the Azure data stack, including ADLS Gen2, Databricks, and Azure Data Factory.
- Working knowledge of Delta Lake, columnar file formats (Parquet, Avro), and data governance.
- Excellent communication skills to partner with cross-functional AI developers and product owners.