Principal Coding Annotator / LLM Evaluation Engineer

About the role

We are seeking an experienced software engineer to join our evaluation and annotation team for state‑of‑the‑art large language models (LLMs). This six‑month contract (with possible extension) focuses on designing coding challenges, assessing model outputs, and feeding insights back into model improvement pipelines.

Key responsibilities

• Create high‑quality coding prompts and reference solutions similar to benchmark suites.
• Evaluate LLM‑generated code for tasks such as generation, refactoring, debugging, and implementation.
• Identify, document, and analyse model failure modes, edge cases, and reasoning gaps.
• Conduct head‑to‑head comparisons between private Mistral‑based LLMs and leading external models.
• Build or configure coding environments to support evaluation and reinforcement‑learning workflows.
• Follow detailed annotation and evaluation guidelines to ensure consistency.

Required profile

• 10+ years of professional software development experience.
• Strong Python programming skills (required) and knowledge of at least one additional language (bonus).
• Minimum 1 year of experience in coding annotation or LLM evaluation, preferably for a frontier AI lab.
• Prior experience as a code reviewer is a plus.
• Fluent written and spoken English.
• Team‑lead or mentoring experience is a strong advantage.

Required skills

• Python
• Code review
• Coding annotation
• LLM evaluation
• Building coding environments

What we offer

• Six‑month contract with the possibility of long‑term extension.
• Opportunity to work hands‑on with cutting‑edge LLMs.
• Flexible location: Paris, London, or remote within Europe for strong candidates.
• Collaboration with a senior, focused technical team.