Principal Data Engineer (Databricks)

Programme: Data Exchange Platform

Industry: Commodities / Energy Trading

Location: London (Hybrid)

Contract: Initial 6-month contract (long-term transformation programme)

X4 Technology are partnered with a global energy trading firm on a major transformation of its Data Exchange platform - the technical data backbone supporting physical commodities trading on a global scale.

The platform automates, reconciles and standardises transaction workflows across physical crude oil and refined petroleum products, giving traders a single, trusted source of truth across the entire ecosystem.

Joining as contract Principal Data Engineer, you'll play a pivotal role in the transformation - combining deep Azure Databricks expertise with genuine technical ownership across data architecture, streaming, integration and platform engineering.

Responsibilities for the Principal Data Engineer (Databricks)

• Own end-to-end data and platform architecture, shaping scalable, secure and high-performance solutions across multiple workstreams
• Lead design/delivery of the Databricks data platform - batch and real-time ingestion, streaming, data modelling, integration across Azure
• Set technical direction and engineering standards, driving best practice on performance, scalability, governance, security, testing and cost optimisation
• Be the go-to across Architects, Engineers, Programme Managers, Business Analysts and Trading stakeholders
• Spot and land pportunities for AI and intelligent automation - MLflow, Azure AI Foundry, Agent Bricks and Microsoft Copilot

Requirements for the Principal Data Engineer (Databricks)

• Extensive hands-on Azure Databricks expertise with a strong track record designing and delivering enterprise-scale data platforms (PySpark, Python, SQL, Delta Lake, Structured Streaming, DLT)
• Proven ownership of data platform architecture - data modelling, integration, governance, security, performance and cost optimisation
• Experience leading complex, greenfield or large-scale data transformation programmes across multiple workstreams and teams
• Strong real-time data and integration experience, including streaming architectures and complex data flows (Kafka beneficial)
• Excellent technical stakeholder management and influencing

High-profile seat at the heart of a global trading data ecosystem. You'll combine hands-on data engineering with real technical influence, shaping a highly secure, cloud-native data platform that connects trading desks, physical assets and external partners

Initial 6-month contract with multi-year programme scope, £850-950/day (inside IR35), hybrid in London. Interviews are being scheduled immediately, with a view to onboarding as soon as possible.