Senior Data Engineer

Senior Data Engineer | Azure Databricks | SC Cleared | London (Hybrid, 3 days on site)

Build the data platform behind national economic analysis.

We are hiring a Senior Data Engineer to develop and scale an Azure Databricks platform that powers monetary analysis, economic forecasting and modelling. The data you engineer feeds the models that inform decisions at a national level, so quality, lineage and reliability genuinely matter here.

This is hands on engineering, not oversight. You will own pipelines end to end, from ingestion through transformation to governed, production ready datasets used daily by economists and data scientists.

What you will be doing

• Designing and building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks
• Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation
• Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into every pipeline
• Implementing governance in practice: Unity Catalog, Azure Purview, data lineage, metadata and monitoring
• Automating deployments through CI/CD in Azure DevOps and promoting DevOps practice across the team
• Working directly with economists and data scientists to turn analytical requirements into engineered solutions

What we need from you

• 8+ years in data engineering, including 3+ years hands on with Azure Databricks
• Strong Python and PySpark, backed by advanced SQL
• Deep grounding in data warehousing, data modelling and integration patterns
• Unity Catalog and Delta Lake in a production environment
• Data quality and governance experience, ideally including Azure Purview
• Git, Agile delivery and a collaborative approach to code review
• Active SC clearance. This is a firm requirement and we are unable to sponsor or process new clearance for this role

Nice to have

• Streaming experience with Kafka or Azure Event Hubs
• Azure Data Engineer Associate or Databricks certification
• Docker, Kubernetes, Terraform or infrastructure as code

Why this role

You get a modern, well funded Azure estate, a mandate to build it properly rather than patch it, and work with real public significance. Hybrid working, three days on site in central London.