Senior AI Solution Architect
Summary
Designs and integrates AI/ML solutions into a data lakehouse using Spark, Kafka, and Jupyter Enterprise Gateway for batch and real-time inference.
Our client is a custom software development company that helps businesses streamline operations and accelerate growth through modern technology solutions. They specialize in building scalable applications tailored to client needs.
Role Introduction
Senior AI Solution Architect will design AI/ML solution patterns that integrate into the existing data platform, leveraging Jupyter Enterprise Gateway, Spark, and Kafka for both real-time and batch inference scenarios.
Features
- Onsite
Requirements
- Design end-to-end AI/ML solution architecture that plugs into the existing Iceberg/Trino/Spark lakehouse.
- Define patterns for both batch inference (via Spark) and real-time inference (via Kafka-based streaming).
- Architect the integration between Jupyter Enterprise Gateway-based development environments and production inference infrastructure.
- Evaluate and select ML serving, feature engineering, and model lifecycle tooling appropriate for the platform.
- Partner with the Data Architect to ensure AI/ML workloads don't conflict with core data platform SLAs or resource allocation.
- Define reference architectures and reusable patterns for common AI use cases relevant to the client's domain.
- Set standards for model governance, monitoring, and explainability appropriate to the deployment context.
- Present architectural recommendations to both technical teams and client stakeholders.
Specifications
- 8+ years in software/data engineering with at least 5 years specifically in ML/AI solution architecture (not just model building).
- Hands‑on experience designing both batch and real‑time inference architectures using Spark and Kafka respectively.
- Familiarity with Jupyter Enterprise Gateway or similar multi‑tenant notebook infrastructure in an enterprise setting.
- Strong understanding of how AI/ML workloads integrate with an existing data lakehouse (Iceberg/Trino/Spark) without disrupting core platform operations.
- Experience with at least one major ML framework (PyTorch or TensorFlow) at an architectural/integration level.
- Strong stakeholder communication skills — this role bridges data platform, ML engineering, and client leadership.
Expertise
- Batch Inference Patterns
- Jupyter Enterprise Gateway
- Kafka
- Spark