freehire launches on Product Hunt on 26 August.

Follow →

codeninja

Discussion

Software Development Engineer in Test

Open 44d

About the Company
CodeNinja is a global technology and AI firm with 10+ years in the industry, helping enterprises modernize systems and embed AI into their workflows. We work across enterprise software, financial services, defense, health, logistics, and more, with 240+ clients across 15 countries. We partner with organizations to deliver AI infrastructure, dedicated engineering teams, and forward-deployed engineers who work directly inside client environments to own outcomes.

About the role

We are hiring an Senior AI QA Automation Engineer an experienced quality engineer who owns feature-level test strategy and automation for AI-enabled products. In this role, you will design and maintain automated tests and AI evaluation suites, integrate quality signals into CI/CD pipelines, and partner with Scrum teams to prevent defects and deliver reliable AI capabilities.

Key Responsibilities

  • Own test strategy and execution for assigned AI features across unit, API, UI, and end-to-end layers, aligned to Definition of Done and release criteria.
  • Create and maintain automated AI evaluation suites (including golden datasets, scoring, and regression baselines) covering key user journeys and failure modes.
  • Validate prompt, retrieval, and orchestration changes using repeatable test harnesses and defined acceptance thresholds.
  • Build and extend automation frameworks, reusable utilities, and improve test stability, speed, and signal-to-noise.
  • Integrate automated tests and AI evaluations into CI/CD, triage pipeline failures, and reduce flaky tests.
  • Partner with engineers to improve testability (contracts, logging, observability) and drive shift-left quality practices.
  • Analyze defects and AI output issues, perform root cause analysis, and implement regression coverage to prevent recurrence.
  • Provide guidance to junior SDETs on automation, test design, and AI validation patterns; communicate quality risks to stakeholders.
  • Support production readiness by validating guardrails, monitoring signals, and performance requirements for AI features.

Requirements

  • Bachelor’s degree in Computer Science/Engineering (or equivalent practical experience).
  • 4+ years of experience in SDET / QA automation / quality engineering with strong hands-on automation and test design skills.
  • Proficiency in at least one programming language (e.g., Python, C#, Java) and SQL, with the ability to debug services and tests.
  • Experience building and maintaining automated tests (API/UI and/or data/ETL) and working in CI/CD pipelines.
  • Experience testing AI-enabled systems or demonstrated applied AI fluency (prompt/RAG changes, evaluation basics, common failure modes).
  • Experience working in Agile/Scrum teams and collaborating effectively with Engineering and Product.
  • Preferred: Experience with LLM evaluation techniques (golden sets, rubric-based grading, automated scoring, review workflows).
  • Preferred: Familiarity with AI safety and guardrail testing (grounding, PII handling, injection risks).
  • Preferred: Experience with cloud tooling, containers, observability, and release automation.

Why Join Us

  • Build quality systems for real-world AI-enabled products, including LLM evaluations and reliability guardrails.
  • Own feature-level quality end-to-end: from test strategy to CI/CD quality gates to production readiness.
  • Work with strong cross-functional teams across Engineering, Product, and AI stakeholders in an Agile environment.
  • Grow in modern quality engineering practices: observability, automation frameworks, shift-left testing, and AI evaluation patterns.

Disclaimer: CodeNinja is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, & performance. Female and minorities are strongly encouraged and preferred to apply for the role

Benefits

    • Provident Fund (PF)
    • Medical Insurance (Pakistan): Coverage includes spouse and parents up to the age of 80.
    • Life Insurance
    • Performance-Based Increment
    • 32 Paid Leaves (Annual)
    • Easy Loan Facility
    • Company-Sponsored Trips
    • Gym Passport Membership (Access To 250+ Gyms)
Show

What this application asks

workable

First name, Last name, Email, Headline, Phone, Address, Photo, Education, Experience, Summary, Resume, Cover letter, What is your current monthly gross salary?, What is your monthly expected gross salary?, What is your relevant number years of experience to this job?, What are your current benefits other than salary?, Please specify your Gender.

  • Could you share an overview of your experience in SDET, QA automation, or quality engineering? Please highlight the types of automation and test design work you've done and how many years you've worked in these areas. written answer
  • How many years of hands-on experience do you have in SDET, QA automation, or quality engineering roles?
  • Can you describe your experience building and maintaining automated tests for APIs, UIs, or data/ETL systems? What tools or frameworks have you used, and what kinds of challenges have you tackled? written answer
  • Do you have experience testing AI/ML systems, especially using MCP (Model Context Protocols)? yes / no
  • What has been your role in integrating automated tests into CI/CD pipelines? Can you share an example of how you've contributed to improving the CI/CD process? written answer
  • In which of the following programming language do you have strong hands-on experience in writing and maintaining test automation code? choose any
  • Have you integrated automated tests into CI/CD pipelines and triaged pipeline or flaky test failures? yes / no
  • Do you have experience testing AI-enabled systems, such as working with prompt engineering, retrieval-augmented generation (RAG), or evaluating AI outputs? Please describe your involvement and any specific techniques or tools you've used. written answer
  • Can you provide an example of a challenging bug or issue you encountered while debugging services or automated tests? How did you identify and resolve it? written answer
  • This position is onsite in DHA Phase 8 Lahore. Will that be possible with you to manage on daily basis? yes / no