AI Quality & Observability Engineer
Description
At Plan.Net CreativeNest, we are a growing creative and technology hub based in South Africa and part of Serviceplan Group, one of Europe's largest independent agency groups.
We bring together creative talent, technology specialists, strategists and engineers to deliver innovative solutions for international clients across the Serviceplan network.
Our AI Engineering team develops and operates advanced multi-agent and Large Language Model (LLM) systems that solve real business challenges. Working closely with our international colleagues, particularly in Germany, we build scalable, reliable AI solutions that are used in production by our clients.
As we continue to grow our AI capabilities, we are looking for an AI Quality & Observability Engineer to take ownership of the quality, reliability and performance of our AI systems.
Your Role
You will be responsible for establishing and maintaining quality and observability standards across our AI engineering projects.
Working alongside Backend Engineers, DevOps specialists and international project teams, you will ensure our AI systems are reliable, measurable, cost-efficient and secure.
You will play a key role in defining how we evaluate AI performance, identify potential issues and continuously improve the systems we deliver to our clients.
What You Will Do
- AI Observability: Implement end-to-end tracing across AI agents, tool calls, model interactions, token usage and latency.
- Quality Assurance: Develop and maintain regression test sets to ensure consistent performance across AI applications.
- LLM Evaluation: Build automated evaluation frameworks, including LLM-as-judge and code-based evaluators, calibrated against human assessments.
- CI/CD Integration: Integrate quality checks into deployment pipelines to identify regressions before production releases.
- Performance Monitoring: Establish dashboards and alerts to monitor system performance, operational costs and latency.
- Incident Management: Investigate quality issues, identify root causes and implement improvements to prevent recurring problems.
- Data Security & Compliance: Develop audit logging and monitoring processes for systems handling confidential or sensitive information.
- Engineering Standards: Define best practices and collaborate with engineering teams to continuously improve the quality of our AI solutions.
What We're Looking For
- Several years of professional experience in Python development and backend engineering, ideally within production environments.
- Hands-on experience building and maintaining evaluation frameworks for production LLM applications.
- Strong understanding of LLM evaluation methodologies, automated scoring and human validation.
- Experience with AWS, particularly CloudWatch and related monitoring services.
- Practical knowledge of observability, including tracing, logging, metrics and alerting, ideally using OpenTelemetry.
- Experience integrating automated quality checks into CI/CD pipelines.
- Strong analytical and problem-solving skills with a proactive approach to quality improvement.
- Excellent written and spoken English communication skills.
- Experience working in international, cross-functional teams, ideally collaborating with colleagues and stakeholders across different countries and time zones.
- Ability to communicate complex technical concepts and quality considerations to non-technical stakeholders.
Nice to Have
- Experience with LangGraph, LangChain or similar agent frameworks.
- Familiarity with Langfuse, LangSmith, Arize Phoenix, Braintrust or comparable evaluation platforms.
- Experience working with AI applications that process confidential or regulated data.
- Knowledge of data protection requirements such as POPIA or GDPR.
- Experience in an agency, consultancy or international technology environment.
Why Join CreativeNest?
At CreativeNest, we combine the agility of a growing South African team with the opportunities of an international agency network.
You will have the chance to help shape our AI engineering capabilities, influence technical decisions and contribute to solutions that are deployed in real client environments.
We value people who take ownership, challenge existing approaches and bring new ideas to the table.
Join us and help build the next generation of reliable, scalable and impactful AI solutions at CreativeNest.
Skills
Sign in to read and join the discussion.