Lead Data Engineer (Kafka/Kinesis)

About the role

We are looking for a Lead Data Engineer to own the design and implementation of a greenfield data platform for an AI‑driven property‑management product. This senior individual‑contributor role focuses on technical leadership rather than people management.

Key responsibilities

• Design and own the company’s data architecture from the ground up.
• Build scalable batch and real‑time data pipelines.
• Develop data ingestion, storage, and transformation frameworks.
• Support AI and machine‑learning infrastructure, including vector search systems.
• Implement data quality, monitoring, and observability practices.
• Collaborate closely with AI, backend engineering, and product teams.
• Optimise performance, scalability, and reliability of large datasets.
• Influence long‑term technical strategy and lay foundations for the data function.

Required profile

• Strong experience in Python development and building scalable distributed data systems.
• Proven expertise in data architecture design and pipeline development for batch and streaming workloads.
• Hands‑on experience with relational databases such as PostgreSQL.
• Familiarity with NoSQL and modern vector databases (e.g., Qdrant, Milvus, pgvector).
• Experience working with Kafka or Kinesis for real‑time data streaming.

Required skills

• Python
• Kafka
• Kinesis
• PostgreSQL
• Qdrant
• Milvus
• pgvector

What we offer

• Salary up to £100,000 per year, dependent on experience.
• Hybrid working model with four days onsite in London.