Databricks Solutions Architect
Posted Updated
Description
Insight Global is seeking a senior data engineer to design, build, and support scalable data and analytics solutions on Databricks. You will develop pipelines, Delta Lake tables, notebooks, workflows, and Databricks SQL assets to empower business users.
The role emphasizes ML model development, AI tooling integration, and collaboration with U.S. teams, requiring strong Python/SQL skills, Databricks expertise, and a collaborative mindset.
Design, build, and support scalable data and analytics solutions on Databricks. Develop data pipelines, Delta Lake tables, notebooks, workflows, and Databricks SQL assets. Create data models, forecasting capabilities, scenario analysis, and reporting solutions. Integrate Databricks with enterprise applications, databases, APIs, cloud storage, and BI tools. Develop and operationalize machine learning models for predictive analytics. Evaluate and implement generative AI solutions using Claude, OpenAI, Azure OpenAI, or similar tools. Build AI-enabled applications using LLM APIs, retrieval-augmented generation (RAG), embeddings, and vector search. Partner with business and technical stakeholders to translate requirements into production-ready solutions. Establish standards for security, governance, testing, documentation, deployment, and performance optimization. Provide technical guidance and support for Databricks, AI, and ML initiatives. Availability to work during U.S. business hours and collaborate effectively with U.S.-based business and technical teams. 5+ years in data engineering, analytics, software development, or data architecture. 3+ years hands-on Databricks experience. Strong Python, SQL, PySpark, and Spark performance‑tuning skills. Experience with Delta Lake, Databricks Workflows, data modeling, and cloud‑based data platforms. Hands‑on experience building and deploying ML models, preferably using MLflow and MLOps practices. Experience with generative AI, LLM APIs, prompt engineering, RAG, embeddings, and vector search. Familiarity with AI tools such as Claude, OpenAI, Azure OpenAI, or similar platforms. Strong communication, problem‑solving, and stakeholder‑management skills. Experience with Azure Databricks, AWS, or GCP. Experience integrating data platforms with enterprise applications and BI tools. Knowledge of Unity Catalog, CI/CD, Git, Terraform, and data governance. Databricks, cloud, or AI/ML certifications.Skills
- AI
- Analytics
- API
- AWS
- Azure
- CI/CD
- Cloud
- Data Engineering
- Data Governance
- Data Modeling
- Data Pipelines
- Databricks
- Delta Lake
- Embeddings
- GCP
- Generative AI
- Git
- LLM
- Machine Learning
- MLflow
- MLOps
- OpenAI
- Predictive Analytics
- Prompt Engineering
- PySpark
- Python
- RAG
- Spark
- SQL
- Terraform
- Unity
- Unity Catalog
- Vector Search
Sign in to read and join the discussion.