Databricks Technical Lead - Azure Platform Migration
Method Resourcing
Job description
About the role
A consulting partner has asked Method Resourcing to find a hands-on Databricks Technical Lead to design and deliver a new Azure Databricks platform for a major life sciences organisation, replacing its enterprise data and analytics stack. The programme uses AI and LLM tooling to reverse-engineer existing pipelines, business logic and reports into documented, tested code. You will mentor three senior data engineers, own the technical architecture and stay hands-on in development. The role is remote first, with occasional on-site work in the Midlands and South East.
Key responsibilities
- Architect and build the platform on Azure Databricks (Delta Lake, Unity Catalog, medallion architecture)
- Use AI agents and LLMs to reverse-engineer legacy reports, queries and transformations, and build repeatable AI-assisted processes for code generation, testing and migration
- Build automated testing and reconciliation to validate migrated data against existing reporting
- Integrate SAP, Azure Data Lake, Salesforce and other systems, and design gold-layer models for Power BI
- Set engineering standards (Workflows, CI/CD, IaC, governance) and mentor the team through to UAT and cutover
Required profile
- Extensive hands-on experience delivering and technically leading enterprise solutions on Azure Databricks
- Demonstrable use of AI, LLMs or agents to reverse-engineer data pipelines or reporting logic
- Proven team leadership while staying hands-on
- Desirable: Databricks certification (Data Engineer Professional or Solutions Architect), consulting background, reusable AI agent frameworks for data engineering, strong governance and data quality experience, platform migration or carve-out experience
Required skills
- Strong Python, PySpark and SQL
- Delta Lake, Unity Catalog, Workflows and medallion architecture
- Azure data engineering (ADLS Gen2, ADF)
- Databricks CI/CD (Terraform, Asset Bundles or Azure DevOps)
- SAP ECC or S/4HANA integration
- Dimensional modelling and Power BI semantic models
The contract is 12 months, outside IR35, at up to £1,000 per day, with some travel required. Candidates apply by sending their CV to the recruiter named in the advert.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in the United Kingdom.
Salaries by job title
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 7 hours ago
Expires 1 month from now
3 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
Method Resourcing
Related job offers
-
Data Engineer – Spécialiste CRM (H/F)
We Are Aspire Dubai City of London -
Data Lead – Responsable de la stratégie data
Reconomy Stevenage -
Senior IT Analyst – Expert Technique et Support (Stratford‑upon‑Avon)
Reconomy Stratford-upon-Avon -
Senior Product Manager – Technologie & Innovation
Reconomy London -
Team Lead Développement Logiciel – Leonardo UK Ltd (Basildon)
Leonardo UK Ltd Basildon