Senior Data Engineer - Databricks

Role: Senior Data Engineer - Databricks

Location: Remote - some travel to client site in London

Contract: Initial 6 months

Rate: £500 to £540 per day

Clearance: Must be eligible to obtain SC Clearance

This role has been deemed Inside IR35

The Opportunity

We are looking for an experienced Senior Data Engineer to work on complex, large-scale data projects within secure and highly regulated environments.

This is an excellent opportunity for a Data Engineer with strong Databricks, PySpark and modern Lakehouse experience who wants to remain hands-on while taking greater ownership of technical delivery.

You'll work across modern cloud data platforms, designing and delivering scalable data pipelines that transform complex datasets into trusted, reusable data products. Alongside engineering delivery, you'll have the opportunity to influence technical standards, contribute to architectural decisions and mentor other engineers.

The Role

As a Senior Data Engineer, you'll take ownership of the design, development and optimisation of production-grade data solutions using Databricks, Apache Spark and Delta Lake.

Your responsibilities will include:

Designing, building and optimising scalable ETL/ELT data pipelines using Databricks, PySpark and Spark SQL

Implementing Medallion Architecture patterns across data layers

Developing robust data quality, monitoring and alerting processes

Building and maintaining modern Lakehouse solutions

Integrating Databricks with cloud services, data lakes, event streams and data warehouses

Applying Unity Catalogue and wider data governance, security and quality controls

Optimising Databricks workloads for performance, reliability and cost

Supporting CI/CD and DevOps practices across data engineering workflows

Leading technical delivery across data engineering workstreams

Working closely with Data Scientists, Analysts, Architects and business stakeholders

Mentoring junior Data Engineers and helping improve engineering standards across the wider team

The Person

We're looking for an experienced Data Engineer who combines strong hands-on engineering skills with the ability to ...