Research Engineer – Agentic Evaluation

About the role

Callosum is building infrastructure that unifies heterogeneous compute for AI. The Research Engineer, Evals will lead the development of rigorous evaluation methods for agentic systems, ensuring that benchmarks reliably reflect capability and guide product decisions.

Key responsibilities

• Design and implement benchmarks and evaluation suites for multi‑turn, long‑horizon, tool‑using agentic behavior.
• Develop statistical methodology to quantify variance, contamination, and construct validity of evaluation results.
• Red‑team internal evaluations and create sanity checks to catch misleading metrics.
• Transform raw execution traces into structured evidence, including failure taxonomies and behavioural signatures.
• Build durable evaluation and observability infrastructure used across the company.

Required profile

• Proven ability to conduct independent research, formulate hypotheses, run experiments and publish reproducible results.
• Deep hands‑on experience with large language models in agentic settings, including multi‑step tasks and tool use.
• Strong statistical discipline: hypothesis pre‑registration, uncertainty quantification and controlled variable comparisons.
• Solid software engineering skills to write robust experiment code within a shared codebase.
• Excellent communication to translate research findings into actionable guidance.

Required skills

• Experience with LLMs and agentic evaluation.
• Statistical analysis and experimental design.
• Software engineering (coding experiments, codebase integration).

What we offer

• Competitive salary based on experience.
• Equity and ownership opportunities.
• Private healthcare.
• Visa sponsorship and relocation support.
• On‑site work in our London office with full resources.