Senior Azure Data Engineer (Databricks)
Capco · UK - London
Job description
About the role
Capco is hiring a Senior Azure Data Engineer (Databricks) in London on a permanent basis, working in a hybrid setting within its Technology & Engineering practice. You will design, build and deploy scalable and secure data pipelines on Databricks on Azure, partner with clients to turn their data needs into solutions, and contribute engineering best practices across cross-functional teams. The role embraces AI-enabled ways of working with approved AI and automation tools.
Key responsibilities
- Design and develop robust data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog and Azure-native services.
- Build real-time, event-driven solutions using Kafka and Azure Event Hubs, applying AI-enabled engineering approaches where appropriate.
- Develop and maintain CI/CD pipelines with Azure DevOps, Jenkins, GitHub Actions and modern DevOps practices.
- Work with clients and multidisciplinary teams to turn business requirements into production-ready data solutions.
- Advocate clean code, data lifecycle optimisation, engineering best practices and the responsible use of approved AI tools.
Required profile
- Proven hands-on experience with Databricks, including orchestration and enterprise-scale Azure data engineering solutions.
- End-to-end data pipeline delivery across ingestion, transformation and serving layers in modern Data Lakehouse architectures.
- Experience with schema design, GDPR-compliant data solutions, DevOps tooling and CI/CD processes.
- A collaborative approach to engineering, using AI-enabled tools responsibly with human oversight, governance and quality.
Required skills
- Python, PySpark and SQL, with distributed data processing frameworks.
- Bonus points: Scala or Java, Cloudera, Hadoop, Hive, Spark, and experience with sensitive data and data privacy regulations.
What we offer
- Discretionary bonus, pension, health, life and critical illness insurance.
- Mental health support, maternity, adoption, shared parental and other paid leave, and eight backup care sessions.
- Five weeks' annual leave, online GP services and a minimum of 40 hours of training per year with a Business Coach.
- Wellhub (Gympass), travel insurance, Tastecard, season ticket loans, Cycle to Work and dental insurance.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in the United Kingdom.
Salaries by job title
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 3 days ago
Expires 1 month from now
10 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
Capco
UK - London