Lead Data Engineer

Lead Data Engineer (Databricks)

Salary £65,000

Type Full time, permanent

Hours 37.5 per week

Location Remote or Hybrid UK, with occasional Richmond office attendance (a couple of times per month)

Overview

We are looking for a Lead Data Engineer to join the Retail Insight Data and Engineering team and own the architecture and development of our Databricks environment. This is a senior, hands on role where you will design and build scalable data infrastructure that underpins our products, while guiding a small team of engineers and defining platform standards, data architecture and engineering best practices supporting analytics, machine learning and AI across the business.

Key responsibilities

* Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability

* Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP

* Define lakehouse standards, data modelling approaches and governance frameworks

* Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies

* Establish security, access control, data quality and compliance standards

* Optimise performance, scalability and cloud cost management

* Lead and mentor a small team of data engineers, setting technical standards and reviewing code

* Partner with data science, analytics and product teams to ensure the platform supports their work

* Own data quality, governance and observability across pipelines

* Support AI and ML enablement including feature stores and model deployment patterns

* Translate business requirements into technical solutions

* Contribute to platform roadmap, tooling selection and continuous improvement

Required experience

* Strong experience as a senior or lead data engineer building and scaling production data platforms

* Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows

* Python and SQL proficiency

* Infrastructure as code experience using Terraform, Bicep or CloudFormation

* Cloud data platform experience, ideally Azure including Azure Data Factory, ADLS Gen2 and Azure DevOps

* Strong understanding of data modelling, ETL and ELT patterns, distributed processing<b...