Databricks Platform Engineer

Location: Hybrid / Monument St, London EC3R 8AJ, UK
job type: Permanent / Full-time
Sector and subsector: IT | Data Engineer
Salary: Competitive salary

Platform Architecture & Engineering responsibilities:

• Design and implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements
• Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers
• Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's))
• Establish secure, compliant environments including networking (VNet/VPC, Private Link), identity (IAM/Entra ID), data governance (Unity Catalog), and access controls
• Define environment strategies (dev/test/prod), CI/CD pipelines, and release processes for Databricks deployments
• Implement monitoring, logging, cost optimisation, and performance tuning across the platform
• Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads

Client Delivery & Enablement responsibilities:

• Work directly with clients to translate business and technical requirements into scalable platform designs
• Lead technical workshops, architecture sessions, and whiteboarding engagements with client stakeholders
• Support rapid prototyping and proof-of-concept builds within Databricks to demonstrate platform capabilities and accelerate client adoption
• Provide best practice guidance on Lakehouse architecture, data modelling, workload optimisation, and cost management
• Produce high-quality technical documentation including architecture diagrams, architecture decision records (ADRs), runbooks, and deployment guides
• Enable client teams through structured knowledge transfer, training, and platform handover
• Collaborate with data engineers, data scientists, and product teams to ensure successful delivery outcomes

Governance & Security:

• Implement Unity Catalog for centralised data governance, including access control (RBAC/ABAC), data lineage, audit logging, and compliance enforcement
• Apply security best practices across platform design: network isolation, secret management, encryption at rest and in transit, and identity federation
• Ensure platform designs meet client regulatory and compliance requirements (e.g. GDPR, ISO 27001, sector-specific standards)

What success looks like in the role:

• Delivery of robust, secure, and scalable Databricks platforms that meet client performance and cost expectations
• Clear, well-architected solutions that balance flexibility, governance, and operational efficiency
• Strong client relationships built on trust, technical credibility, and effective communication
• Accelerated client adoption of the Lakehouse platform through well-designed enablement and documentation
• Reduced deployment time through reusable infrastructure patterns and automation
• Proactive identification of risks, trade-offs, and optimisation opportunities across platform design and delivery
• Contribution to the organisation’s growing body of reusable platform accelerators, reference architectures, and internal knowledge

Competencies and Behaviours:

• 3+ years experience in data platform engineering, cloud engineering, or similar roles
• Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows
• Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/VPC, IAM)
• Experience with infrastructure-as-code tools (e.g. Terraform preferred) and CI/CD pipelines (e.g. Azure DevOps, GitHub Actions)
• Solid understanding of data architecture concepts including Lakehouse medallion architecture and dimensional modelling
• Familiarity with security and governance frameworks (e.g. RBAC, ABAC, data masking, audit, compliance standards)
• Excellent communication skills with the ability to explain complex technical concepts to non-technical stakeholders
• Comfortable working in a client-facing consultancy environment with multiple concurrent engagements
• Proactive, self-driven, and able to take ownership of end-to-end platform delivery
• Willingness to travel within the UK as required
• Right to work in the UK