Senior Data Engineer

Senior Data Engineer | Azure Databricks | SC Cleared | London (Hybrid, 3 days on site)

Build the data platform behind national economic analysis.

We are hiring a Senior Data Engineer to develop and scale an Azure Databricks platform that powers monetary analysis, economic forecasting and modelling. The data you engineer feeds the models that inform decisions at a national level, so quality, lineage and reliability genuinely matter here.

This is hands on engineering, not oversight. You will own pipelines end to end, from ingestion through transformation to governed, production ready datasets used daily by economists and data scientists.

What you will be doing

Designing and building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks

Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation

Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into every pipeline

Implementing governance in practice: Unity Catalog, Azure Purview, data lineage, metadata and monitoring

Automating deployments through CI/CD in Azure DevOps and promoting DevOps practice across the team

Working directly with economists and data scientists to turn analytical requirements into engineered solutions

What we need from you

8+ years in data engineering, including 3+ years hands on with Azure Databricks

Strong Python and PySpark, backed by advanced SQL

Deep grounding in data warehousing, data modelling and integration patterns

Unity Catalog and Delta Lake in a production environment

Data quality and governance experience, ideally including Azure Purview

Git, Agile delivery and a collaborative approach to code review

Active SC clearance. This is a firm requirement and we are unable to sponsor or process new clearance for this role

Nice to have

Streaming experience with Kafka or Azure Event Hubs

Azure Data Engineer Associate or Databricks certification

Docker, Kubernetes, Terraform or infrastructure as code

Why this role

You get a modern, well funded Azure estate, a mandate to build it properly rather than patch it, and work with real public significance. Hybrid working, three days on site in central London.

#J-18808-Ljbffr