Lead Data Engineer

About the role

As a Lead Data Engineer you will design and deliver mission‑critical data solutions for national security, defence and intelligence customers. Working with cutting‑edge technologies such as AI/DS, cyber, cloud and DevOps, you will translate complex business problems into robust data pipelines and analytics platforms.

Key responsibilities

• Design, build and maintain end‑to‑end data pipelines, including ingestion, orchestration and enrichment.
• Develop streaming and ETL solutions using tools such as Apache NiFi.
• Model databases and integrate data from diverse operational sources.
• Ensure data quality, consistency and security throughout the lifecycle.
• Monitor, tune and optimise system performance.
• Write clean, secure, reusable and test‑driven code.
• Apply systems‑integration expertise within agile teams and decompose user needs into epics and stories.
• Promote reuse of data flows, best practices and data‑engineering standards across government projects.

Required profile

• Active Enhanced DV (West) clearance.
• Willingness to work on‑site in Gloucester when required, with hybrid flexibility.
• Proven experience with Apache Kafka, Apache NiFi, SQL and NoSQL databases (e.g., MongoDB).
• Hands‑on experience in ETL processing using Groovy, Python or Java.
• Ability to interpret technical and business stakeholder needs and communicate clearly.
• Leadership skills to guide teams and manage senior‑level conversations.

Required skills

• Apache Kafka
• Apache NiFi
• SQL
• NoSQL (MongoDB)
• Groovy
• Python
• Java

What we offer

• Competitive salary (GBP 65,000‑85,000 per year) plus 6% bonus.
• Clearance bonus and 25 days holiday.
• Hybrid working where possible.
• Opportunity to work on high‑impact national security projects.