Senior Data Engineer & Technical Delivery Lead (Python/Databricks)

About the role

We are supporting a major financial services organisation that is expanding its data engineering function. The role combines hands‑on data engineering with technical delivery leadership on a large‑scale cloud transformation programme.

Key responsibilities

• Design, develop and maintain scalable data pipelines using Python and Databricks.
• Build, optimise and support ETL/ELT workflows with Apache Spark, PySpark and Delta Lake.
• Create robust data models and Lakehouse architectures.
• Implement and manage data workflows within the Databricks ecosystem, ensuring data quality and reliability.
• Collaborate with architects, product owners, analysts and engineering teams to deliver high‑quality data solutions.
• Provide technical leadership, architectural oversight and end‑to‑end delivery for complex initiatives (Technical Delivery Lead).

Required profile

• Strong commercial experience developing data engineering solutions with Python.
• Hands‑on expertise in Databricks, Spark, Delta Lake and modern Lakehouse design.
• Proven ability to build scalable ETL/ELT pipelines and data models.
• Experience with cloud platforms (AWS, Azure or GCP) and modern data warehouses (Snowflake, Redshift, BigQuery).
• Familiarity with Agile delivery environments.

Required skills

• Python
• Databricks (Workflows, Notebooks, Delta Lake)
• Apache Spark / PySpark
• SQL and data modelling
• Lakehouse architecture
• AWS, Azure, GCP
• Snowflake, Redshift, BigQuery
• Airflow or Databricks Workflows
• Power BI
• React
• Kafka / Structured Streaming
• CI/CD and DevOps practices
• Docker
• Kubernetes
• Machine Learning pipelines