IT Security Engineer (EVAL SOTA)

We are looking for EVAL SOTA Experts to build realistic, Dockerized evaluation environments that test whether state-of-the-art AI coding agents behave safely while completing real software engineering tasks. The goal is to create scenarios where a careful agent can complete the task safely, but strong frontier models may still violate an important safety constraint. All evaluations must be fully code-graded and reproducible, without relying on an LLM judge.

Responsibilities:

• Design realistic coding scenarios with clear safety constraints.
• Build complete offline Docker environments, repository fixtures, instructions, and reference solutions.
• Write deterministic pytest-based verifiers for both task completion and safety behavior.
• Run frontier AI models against the environment and review their trajectories.
• Identify exactly where and why an AI agent violated the safety rule.
• Ensure every evaluation is reproducible and produces a reliable verdict.
• Create adversarial but fair scenarios where unsafe behavior is avoidable.
• Iterate on environments based on structured review feedback.

Code Security:

• Build environments that test issues such as injection vulnerabilities, broken access control, secret leakage, unsafe deserialization, path traversal, SSRF, and weak cryptography.
• Relevant backgrounds: AppSec, secure code review, penetration testing.

Cyber Harm

• Evaluate whether agents engage in malware-related behavior, exploitation, credential theft, data exfiltration, and surveillance.
• Relevant backgrounds: Offensive Security, Red Teaming, CTF.

Data Handling:

• Test whether sensitive data leaves its permitted boundary, is shared without assessing sensitivity, or is mishandled during normal workflows.
• Relevant backgrounds: Privacy Engineering, DLP, Data Platforms.

Autonomy and Control:

• Evaluate whether agents act beyond their assigned scope, ignore or resist oversight, pursue unrequested goals, or attempt to game tests or evaluation systems.
• Relevant backgrounds: SRE, DevOps, Agent Development, AI Safety.

Deception:

• Test whether agents fabricate results, hide errors, or misrepresent task completion; make unsupported claims about their own work.
• Relevant backgrounds: QA, test engineering, AI evaluation.

Requirements:

• Strong hands-on Python development skills.
• Experience writing production-quality code and pytest test harnesses.
• Strong proficiency with Docker and isolated/offline environments.
• Deep expertise in at least one of the five safety dimensions above.
• Ability to design adversarial yet fair evaluation scenarios.
• Strong code review, debugging, and technical reasoning skills.
• Ability to inspect AI-agent trajectories and pinpoint failure points.
• Strong technical documentation and communication skills.

Preferred Skills:

• Red teaming or security research.
• AI evaluation or benchmark development.
• RLHF or model evaluation experience.
• Hands-on work with coding agents and evaluation harnesses.
• Experience with Node.js, Go, or Java in addition to Python.
• CVE research, vulnerability analysis, or strong CTF experience.
• Secure software engineering or penetration-testing background.