Senior Data Engineer & Technical Delivery Lead (Python / Databricks)

About the role

Join a leading investment bank on a large‑scale cloud data transformation programme. As a Senior Data Engineer or Technical Delivery Lead you will work hands‑on with Python, Databricks and Spark to design, build and optimise enterprise‑grade data platforms in a modern Lakehouse environment.

Key responsibilities

• Design, develop and maintain scalable data pipelines using Python and Databricks.
• Build, optimise and support ETL/ELT workflows with Apache Spark, PySpark and Delta Lake.
• Architect robust data models and Lakehouse structures.
• Implement and manage data workflows within the Databricks ecosystem.
• Ensure data quality, governance and pipeline reliability.
• Collaborate with architects, product owners, analysts and engineering teams.
• Implement monitoring, logging and alerting for critical data platforms.
• Provide technical leadership, architectural oversight and end‑to‑end delivery for complex initiatives (Technical Delivery Lead).

Required profile

• Strong commercial experience developing data engineering solutions with Python.
• Extensive hands‑on experience with Databricks (Workflows, Notebooks, Delta Lake).
• Proven expertise in Apache Spark / PySpark and building scalable ETL/ELT pipelines.
• Solid SQL and data‑modelling skills.
• Experience with cloud platforms (AWS, Azure or GCP) and modern data‑warehousing technologies.
• Background in Agile delivery environments.

Required skills

• Python
• Databricks
• Apache Spark / PySpark
• Delta Lake
• SQL
• Lakehouse architecture
• AWS, Azure, GCP
• Snowflake, Redshift, BigQuery
• Airflow or Databricks Workflows
• Kafka
• Structured Streaming
• CI/CD and DevOps practices
• Docker
• Kubernetes
• Machine Learning pipelines