AI Evaluation Engineer

About the role

We are seeking an experienced AI Evaluation Engineer to join a specialist Public Sector Technology team. The role is hands‑on, focusing on building evaluation frameworks, tooling and harnesses for AI systems, especially large language models and agentic AI solutions. You will work directly with government teams to prototype, test and deliver rapid AI evaluations.

Key responsibilities

• Design and develop evaluation frameworks, tooling and repeatable processes.
• Create evaluation harnesses to test model and agentic AI systems.
• Define metrics, methodologies and benchmark AI performance.
• Investigate AI failure modes and present findings with recommendations.
• Collaborate with government stakeholders to prototype and test solutions.

Required profile

• Strong software engineering background with experience building and evaluating AI systems.
• Deep understanding of LLMs, agentic AI and AI evaluation frameworks.
• Ability to work autonomously, solve problems and communicate results effectively.

Required skills

• Python development
• AI/LLM evaluation
• RAG evaluation and Ragas
• Agentic AI evaluation
• Evaluation harness creation
• LLM testing and benchmarking
• Prompt engineering
• Model and agent integration