Lead Databricks Data Engineer – £500‑£600/day (Hybrid, London)

About the role

We are seeking a Lead Databricks Data Engineer to shape the future‑state data platform for a large, complex financial services organisation. The role focuses on moving from slow, batch‑based analytics to a modern, streaming‑led architecture while owning the data vision, use cases and delivery direction.

Key responsibilities

• Define a clear end‑to‑end data vision, covering current state, target architecture and practical migration steps.
• Lead the design and implementation of a Databricks‑based medallion architecture, with a strong emphasis on a robust Gold layer for analytics.
• Shift data availability from monthly batch processes to more frequent or streaming ingestion where appropriate.
• Act as the senior technical decision‑maker on data use cases, prioritisation and trade‑offs.
• Collaborate with onshore teams and an offshore partner to unblock progress, clarify ownership and accelerate delivery.
• Provide regular, concise updates to senior stakeholders to ensure alignment.

Required profile

• Proven experience as a Lead Data Engineer within financial services, preferably pensions or other regulated environments.
• Demonstrated ability to design and deliver modern data platforms using Databricks and medallion architecture principles.
• Experience handling large volumes of legacy data and integrating it with contemporary analytics platforms.
• Strong capability to translate business needs into technical delivery and to lead mixed onshore/offshore teams.
• Excellent communication skills with credibility to engage senior stakeholders.

Required skills

• Databricks
• Medallion architecture
• Streaming data ingestion
• Batch processing
• Data governance

What we offer

• Day rate of £500‑£600, outside IR35.
• Hybrid working arrangement based in London.
• Freelance contract with the opportunity to influence a strategic data platform.