Senior Data Scientist

We're looking for a Senior Data Scientist to join our AMI team. This hands-on role combines Machine Learning, statistical modeling, Generative AI, RAG, model evaluation, and fine-tuning for enterprise AI applications. AMI supports commercial AI providers and on-premises models, with agents, MCP tools, and A2A interoperability delivered in partnership with AI Engineering.

About the Role:

You will independently design, develop, evaluate, and operationalize complex AI/ML solutions. You will contribute hands-on technical expertise, apply rigorous experimentation and MLOps/LLMOps practices, improve modeling and evaluation approaches within assigned initiatives, and mentor other Data Scientists.

Machine Learning & Applied Data Science

• Apply classification, regression, ranking, forecasting, clustering, and anomaly detection to enterprise AI and agentic use cases.
• Develop and evaluate intent classification, semantic matching, agent/skill/tool ranking, and model-routing techniques; compare learned approaches with rules, embeddings, and LLM-based baselines.
• Use statistical modeling to estimate execution latency, token demand, inference costs, workload trends, and resource requirements.
• Apply clustering and anomaly detection to identify recurring intents, capability gaps, unusual agent behavior, and execution regressions.
• Conduct exploratory analysis, dataset preparation, feature engineering, hypothesis testing, and reproducible experiments.

Foundation Models, Prompting & Context Engineering

• Evaluate commercial API and self-hosted open-weight models for accuracy, reasoning, instruction adherence, structured outputs, latency, throughput, and total cost.
• Design and optimize system instructions, prompt templates, few-shot examples, tool-calling prompts, and context assembly strategies.
• Benchmark model alternatives for specific enterprise workloads and document quality/cost/latency tradeoffs.
• Collaborate with AI Engineering and Platform Engineering on inference performance, quantization tradeoffs, and GPU/VRAM capacity assumptions.

Retrieval-Augmented Generation (RAG)

• Design and optimize document parsing, chunking, metadata, embedding, retrieval, reranking, and context selection strategies.
• Evaluate vector, keyword/BM25, hybrid, and semantic retrieval, including access-aware retrieval where relevant.
• Diagnose retrieval versus generation failures; measure Recall@K, nDCG, groundedness, faithfulness, and end-to-end answer quality.
• Compare embedding and reranking models using representative enterprise datasets and reproducible benchmarks.

Model Fine-Tuning & Adaptation

• Assess when prompting, RAG, supervised instruction fine-tuning, or parameter-efficient adaptation is appropriate.
• Curate instruction datasets and holdout evaluations; apply SFT, LoRA, QLoRA, or comparable methods.
• Measure gains over base-model and prompting/RAG baselines, including generalization, regressions, data leakage, and operating costs.
• Collaborate on the packaging and validation of fine-tuned model artifacts and adapters.

Agentic AI, MCP & A2A

• Apply modeling and evaluation expertise to intent routing, agent/skill discovery, MCP tool selection, workflow selection, and planning effectiveness.
• Evaluate task completion, tool-call correctness, argument extraction, delegation quality, and recovery from failures.
• Work with AI Engineering on agentic frameworks and understand MCP tool schemas and A2A Agent Cards, skills, and task delegation.
• Contribute decision models and quality signals without owning agent runtime, MCP server, or A2A protocol implementation.

AI Evaluation & Experimentation

• Design golden datasets, holdout sets, human review protocols, LLM-as-judge calibration, and statistical comparison methods.
• Automate evaluations and regression checks for prompts, foundation models, RAG, fine-tuned models, routing, and agents.
• Define measurable quality thresholds for accuracy, relevance, groundedness, instruction following, task completion, and cost/performance.
• Investigate regressions and use evidence to guide model, prompt, retrieval, and routing decisions.
• Implement and maintain reproducible experimentation, model evaluation, and lifecycle workflows for assigned AI/ML initiatives, including automated validation and production readiness checks.

MLOps, LLMOps & Model Lifecycle

• Apply reproducible experiment tracking across datasets, code, model versions, hyperparameters, prompts, retrieval configurations, metrics, and artifacts.
• Develop and maintain automated training, fine-tuning, benchmarking, and evaluation pipelines, with model registry and artifact traceability.
• Define and integrate quality gates, regression tests, and model/prompt/RAG release-readiness criteria with engineering teams.
• Analyze production quality and drift signals, agent routing accuracy, latency, throughput, and token usage; recommend remediation.
• Collaborate with engineering on packaging, deployment handoffs, observability, rollback criteria, and ongoing model lifecycle management.

Technical Leadership & Collaboration

• Independently lead complex Data Science initiatives from problem formulation through model development, evaluation, and production handoff.
• Make technical decisions within assigned initiatives, balancing model quality, performance, cost, and delivery requirements.
• Improve reusable modeling, benchmarking, experimentation, and MLOps/LLMOps practices within assigned initiatives.
• Collaborate with AI Engineering, AI Quality, Platform Engineering, Product Management, and domain experts.
• Translate business requirements into measurable AI and ML objectives.
• Mentor junior and mid-level Data Scientists and contribute to technical reviews.
• Communicate complex AI and ML concepts clearly to technical and non-technical stakeholders.

Must-Have Skills & Experience

• 8+ years of relevant Data Science, Machine Learning, or Applied AI experience.
• Bachelor’s degree in Machine Learning, Statistics, Data Science, Physics, Computer Science, or a related field, or equivalent education and industry experience.
• Advanced Python proficiency with Pandas, NumPy, scikit-learn, and PyTorch or TensorFlow; SQL and practical data preparation skills.
• Strong statistics, experimental design, hypothesis testing, ML validation, and applied classification, regression, ranking, clustering, forecasting, and anomaly detection knowledge.
• Hands-on LLM prompting, embeddings, model comparison, and RAG design/optimization, including chunking, retrieval, and reranking.
• Practical instruction fine-tuning or parameter-efficient model adaptation experience, with measurable baseline comparisons.
• Experience designing evaluation datasets, automated quality metrics, and statistical model/LLM/RAG experiments.
• Hands-on MLOps and LLMOps: experiment tracking, model/dataset/prompt versioning, reproducible pipelines, registries/artifacts, evaluation automation, quality gates, and monitoring.
• Working knowledge of agent planning, routing, tool calling, MCP, A2A, and multi-agent patterns.
• Understanding of inference latency, throughput, concurrency, token cost, and on-prem versus provider-hosted model tradeoffs.
• Familiarity with AWS and/or Kubernetes or Red Hat OpenShift (OCP), and collaboration with platform engineering.
• Clear communication, independent problem solving, cross-functional collaboration, and mentoring.
• Demonstrated ability to independently deliver complex AI/ML solutions, make sound technical decisions within assigned initiatives, and mentor other Data Scientists.
• Experience contributing to reusable modeling, evaluation, and MLOps/LLMOps practices within project teams.
• Evidence We Will Look for in Hiring
• Delivered measurable improvements in model quality, RAG retrieval, routing accuracy, fine-tuning outcomes, or inference cost against documented baselines.
• Built reproducible ML/LLM experiments, automated evaluation pipelines, tracked artifacts, and model quality gates.
• Demonstrated independent delivery of complex AI/ML solutions, rigorous model validation, collaboration on production monitoring, mentoring of other Data Scientists, and improvements to technical practices within assigned initiatives.
• Demonstrated improvements in model quality, reliability, performance, or operational efficiency within specific projects or applications.

Nice-to-Have Skills

• Experience with OpenSearch, Elasticsearch, FAISS, Milvus, pgvector, or similar retrieval platforms.
• Hands-on use of LangGraph, OpenAI Agents SDK, Google ADK, CrewAI, or other agentic frameworks.
• Familiarity with MLflow, Kubeflow, Ragas, DeepEval, OpenJudge, LangSmith, or comparable MLOps/LLMOps and evaluation tooling.
• Experience with vLLM, NVIDIA Triton, quantization, GPU memory sizing, or distributed inference.
• Python backend APIs (FastAPI, Flask, or similar); OpenTelemetry and AI observability.
• AI safety, prompt injection evaluation, privacy, responsible AI, and model governance.
• Enterprise code understanding, legacy modernization, or domain-specific model adaptation.
• Agile practices and Atlassian Jira/Confluence.
• Master’s degree in a related field.