Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

dataSpecta

NewBe an early applicant

Senior AI QA / Evaluation Engineer

Posted

Description

About DataSpecta


DataSpecta is a Data & AI consulting and engineering company helping organizations transform data into measurable business value through advanced analytics, modern data platforms and enterprise AI solutions.


We design and deliver solutions across both on-premise and cloud environments, working closely with enterprise customers from discovery through production.


Alongside customer projects, we build our own AI products such as CogniSpecta (enterprise knowledge & cognitive search) and CodeSpecta (AI-powered software engineering).


We're looking for a Senior AI QA / Evaluation Engineer based in Baku, Azerbaijan, with strong test automation expertise and hands-on experience evaluating LLM and ML systems.


You will own quality assurance for production AI systems in a regulated enterprise environment, acting as the quality gate for every production release.


Location & Work Model

  • Location: Baku, Azerbaijan
  • Work model: [Hybrid / On-site]


What You Will Work On

  • Evaluation harnesses and golden datasets for enterprise AI agents
  • Evaluation of RAG assistants and report generation agents
  • Adversarial and security testing for LLM applications
  • Audit evidence and quality governance


Responsibilities

  • Build versioned golden datasets and evaluation harnesses
  • Gate production releases on evaluation baselines in CI
  • Revalidate agents at each model version change
  • Run quality control on recurring reports
  • Run adversarial tests for prompt injection, data leakage and unsafe tool use
  • Triage support and agent output incidents
  • Maintain the audit evidence pack


Required Qualifications

  • Based in Baku or willing to relocate
  • 4+ years in QA or test engineering, including test automation ownership
  • Proven experience evaluating LLM or ML outputs
  • Strong Python skills for evaluation tooling
  • Experience with formal defect management
  • Fluent English


Technical Skills

  • Evaluation frameworks: Ragas, DeepEval, promptfoo or equivalent
  • Metrics: Groundedness, faithfulness, context precision/recall, answer correctness, tool call accuracy
  • Statistics: Repeated sampling, bootstrapped confidence intervals
  • Automation: Python, pytest, API testing, Playwright, CI
  • Adversarial testing: OWASP Top 10 for LLMs, garak, PyRIT
  • Observability: Langfuse, Arize Phoenix or equivalent
  • Multilingual: Azerbaijani and mixed-language evaluation


Preferred Qualifications (a plus)

  • Audit evidence experience in a regulated sector
  • Model risk management frameworks


What We Value

  • Curiosity about solving business problems with AI
  • Comfort across business, data and technology
  • End-to-end ownership
  • Continuous learning


Follow us:

  • LinkedIn: https://www.linkedin.com/company/innovance-consultancy
  • LinkedIn: https://www.linkedin.com/company/dataspecta
  • Instagram: https://www.instagram.com/innovanceconsultancy
  • Instagram: https://www.instagram.com/dataspecta

Skills

What Senior QA jobs ask for — and how much of it you have →
Apply

See also

QA jobs by country — openings, pay and top skills →