Lead Backend Software Engineer – Data Services (Contract)

About the role

We are seeking a Lead Backend Software Engineer to join a high‑profile programme building secure, cloud‑native data services that enable data‑driven research at national scale. This founding engineering role will shape the architecture, standards and core services of a federated data platform operating in a highly regulated environment.

Key responsibilities

• Design and develop scalable, cloud‑native data services for a federated platform.
• Define architectural patterns, engineering standards and best practices across the organisation.
• Build distributed server‑side services using modern software engineering practices.
• Architect federated data services that securely query and process data across multiple environments.
• Create high‑performance data processing and streaming workflows with Apache Spark, Apache DataFusion or similar frameworks.
• Implement Kubernetes‑based services and containerised workloads.
• Develop secure APIs supporting authentication, authorisation, governance and audit requirements.
• Establish CI/CD pipelines, Infrastructure as Code, automated testing and deployment practices.
• Mentor engineers and promote engineering excellence.

Required profile

• Proven experience in a senior engineering role such as Lead Engineer, Principal Engineer, Staff Engineer or Software Architect.
• Comfortable working from first principles in ambiguous, fast‑moving environments.
• Strong ability to balance strategic architectural thinking with hands‑on software development.
• Experience delivering complex distributed systems in regulated sectors.

Required skills

• Python
• Apache Spark
• Apache DataFusion (or similar distributed query framework)
• Kubernetes
• CI/CD pipelines
• Infrastructure as Code

What we offer

• Rate £700–£800 per day (outside IR35)
• Initial 6‑month contract with strong likelihood of extension
• Hybrid work model – 2 days per week onsite in London