Lead Azure Databricks Platform Engineer / Architect

Lead Azure Databricks Platform Engineer / Architect

Hands-on Platform Engineering | Serverless | FinOps | POSIT/RStudio Migration

6 Month contract

Inside IR35 - 500 a day

London/Hybrid

Role Purpose

We are seeking a highly experienced, hands-on Azure Databricks Platform Engineer / Architect to enhance and optimise an enterprise Data Platform. The role combines architecture with direct implementation: the successful candidate must be able to configure, develop, troubleshoot and optimise Azure Databricks rather than operate only at design or governance level.

The role is centred on three outcomes: enabling and optimising Databricks Serverless, strengthening FinOps and platform controls, and enhancing the Databricks Discovery Zone to support workloads currently delivered through POSIT/RStudio.

Key Responsibilities

1. Databricks Serverless Enablement and Optimisation

Assess existing workloads and determine suitability for Serverless, classic, job or interactive compute based on duration, utilisation, SLA, concurrency, performance and cost.

Enable and configure Serverless for appropriate jobs, SQL workloads, notebooks, analytical processing and data pipelines.

Establish workload-placement guidance, including when Serverless is not economical for predictable, heavy or continuously running workloads.

Implement compute policies, autoscaling, quotas, budget controls and operational guardrails.

Measure cost and performance outcomes, identify idle or oversized compute, and recommend optimisation actions.

2. FinOps and Enterprise Platform Controls

Define and embed a practical FinOps operating model covering ownership, accountability, projects, environments, teams, applications and cost centres.

Implement mandatory tagging and integrate validation into CI/CD so non-compliant resources are prevented from being provisioned.

Provide granular cost attribution by workspace, project, application, workload, job and team/user where technically appropriate.

Implement budget policies, thresholds, proactive alerts and usage reporting to prevent uncontrolled spend.

Use platform usage and billing data to identify idle compute, inefficient workloads, unnecessary storage/data movement an...