Research Engineer – Benchmarking (Member of Technical Staff)

About the role

Callosum is building a unified benchmarking system to evaluate multi‑step LLM and agentic solutions. The role combines rigorous research evaluation with production‑grade engineering to create reproducible, contamination‑controlled benchmarks that guide product decisions.

Key responsibilities

• Design and implement a unified platform that measures task success, quality, and robustness across diverse agent topologies and decomposition strategies.
• Mine real code commits and execution traces, run sandboxed grading, and construct task suites reflecting real‑world workloads such as code search, edit, repair, and repository summarisation.
• Enforce strict controls against benchmark contamination, overfitting, and metric gaming while keeping baselines stable over time.
• Compare algorithmic and agentic approaches honestly, feeding results directly into product shipping decisions and customer proof points.
• Lead external benchmark co‑publications and ensure they survive peer and customer review.

Required profile

• PhD in computer science, machine learning, or a related field, or an equivalent research track record.
• Authorship or co‑authorship of a benchmark or evaluation paper at a recognised venue (e.g., NeurIPS, ICML, ICLR, ACL).
• Deep understanding of pitfalls in LLM and agent evaluation such as contamination, overfitting, and weak baselines.
• Proven ability to design, build, and curate large‑scale evaluation systems with strong Python and CI skills.

Required skills

• Python programming.
• Sandboxed and distributed execution environments.
• Continuous integration (CI) pipelines.
• Experience building or rigorously evaluating multi‑step LLM and agentic systems.

What we offer

• Competitive salary determined by skills and experience.
• Equity and ownership opportunities.
• Private healthcare.
• Visa sponsorship and relocation support.