Sr AI Engineer
Description
Role: Sr AI Engineer
Location:Onsite. Chicago, IL
Type: C2C
Send profiles to
Job Description:
We are seeking a Senior AI Engineer with deep, hands-on expertise in Databricks to design, build, and deploy production-grade AI and machine learning solutions at scale. You will lead the end-to-end lifecycle of data-intensive AI applications—from building resilient, large-scale data ingestion and transformation pipelines to training, fine-tuning, and operationalizing ML and Generative AI models in a unified lakehouse environment.
Key Responsibilities:
Large-Scale Data Engineering & Transformation: Architect and optimize high-throughput batch and streaming ETL/ELT pipelines using Apache Spark, PySpark, Delta Lake, and Databricks Lakehouse architectures to process multi-terabyte datasets.
AI & Machine Learning Model Development: Design, train, evaluate, and fine-tune predictive ML models, deep learning architectures, and Generative AI/LLM workflows (e.g., RAG pipelines, agentic systems) tailored to business-critical use cases.
MLOps & Production Deployment: Operationalize models at scale using MLflow, Databricks Model Serving, and CI/CD best practices, ensuring automated retraining, drift monitoring, and low-latency inference.
Data Governance & Feature Engineering: Build reusable feature pipelines using Databricks Feature Store and enforce data quality, lineage, and access controls via Unity Catalog.
Performance & Cost Optimization: Tune Spark jobs, cluster configurations, Photon engine workloads, and Delta table partitioning/Z-ordering to maximize compute efficiency and minimize cloud infrastructure costs.
Cross-Functional Technical Leadership: Partner with data scientists, software engineers, and product managers to translate complex business requirements into scalable technical architectures, while mentoring junior engineers on coding and MLOps standards.
Required Qualifications:
Experience: 5+ years of hands-on experience in AI/ML engineering, data science, or big data engineering, with at least 3+ years building production systems on Databricks.
Databricks Ecosystem Mastery: Proven expertise in PySpark, Spark SQL, Delta Lake, MLflow, Unity Catalog, and Databricks Workflows.
Programming & Software Engineering: Strong proficiency in Python and SQL, with solid software engineering fundamentals (Git, modular design, unit testing, CI/CD).
AI/ML Frameworks: Deep familiarity with modern ML and deep learning libraries (e.g., PyTorch, TensorFlow, Scikit-learn, Hugging Face, LangChain).
Cloud Infrastructure: Hands-on experience deploying data and AI workloads on major cloud platforms (AWS, Azure, or Google Cloud Platform).
Preferred Qualifications:
Experience building and deploying Generative AI / LLM applications using Databricks Mosaic AI, Vector Search, and Retrieval-Augmented Generation (RAG).
Familiarity with infrastructure-as-code tools (Terraform, Databricks Asset Bundles) for automated environment provisioning.
Experience with real-time streaming technologies (Apache Kafka, Spark Structured Streaming).
Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, or a related quantitative field.
Skills
- Agentic AI
- AI
- AWS
- Azure
- Big Data
- CI/CD
- Cloud
- Data Engineering
- Data Governance
- Data Ingestion
- Data Quality
- Data Science
- Databricks
- Deep Learning
- Delta Lake
- ELT
- ETL
- Feature Engineering
- Fine Tuning
- GCP
- Generative AI
- Git
- Hugging Face
- Infrastructure as Code
- Kafka
- Lakehouse
- LangChain
- LLM
- Machine Learning
- MLflow
- MLOps
- PySpark
- Python
- PyTorch
- RAG
- scikit-learn
- Spark
- SQL
- TensorFlow
- Terraform
- Unit Testing
- Unity
- Unity Catalog
- Vector Search
Sign in to read and join the discussion.