Lead Databricks Engineer

We’re looking for a Lead Databricks Engineer to join our team in London, UK in a hybrid working mode. This role involves designing and leading enterprise-scale Azure and Databricks data platforms to enable advanced analytics and real-time processing capabilities. You will be responsible for platform architecture, governance, cost optimization and ensuring delivery of high-quality solutions that meet critical business needs in a complex, data-driven environment.ResponsibilitiesLead architecture design and implementation of scalable solutions on Azure DatabricksDefine and enforce platform standards, ensuring compliance with data governance and security policiesBuild and optimize Databricks streaming workloads, including Structured Streaming and Delta Live TablesCollaborate with cross-functional teams to translate requirements into robust technical solutionsDrive platform performance tuning, cost optimization and monitoring strategies for cloud workloadsProvide leadership across the full engineering lifecycle, including solution delivery and production supportMentor and guide developers, establishing best practices for data engineering on Azure and DatabricksParticipate in technical decision-making and review designs for scalability and maintainabilityImplement observability for critical pipelines and maintain quality assurance standardsEngage directly with stakeholders to ensure alignment on strategic platform initiativesRequirementsMinimum 8+ years of experience in data engineering, including 3+ years in a tech lead capacityStrong hands-on expertise in Databricks for enterprise-scale workloadsProficiency in PySpark, Spark Structured Streaming, and Delta LakeSolid programming skills in Python and demonstrated SQL performance optimization experienceDeep understanding of Azure Data Platform services and cloud-native engineering patternsProven experience implementing data governance, quality management and observability frameworksAbility to manage complex data pipelines for batch and streaming use cases in mission-critical environmentsStrong communication and leadership skills to work with distributed teams and senior stakeholdersNice to haveKnowledge of Delta Live Tables (DLT) and advanced workflows in DatabricksFamiliarity with CI/CD for data engineering using GitHub or Azure DevOpsExperience designing infrastructure-as-code solutions for Azure environmentsUnderstanding of data mesh or lakehouse principles for large-scale architecturesBackground in financial trading or capital markets data domainsWe offerEPAM Employee Stock Purchase Plan (ESPP)Protection benefits including life assurance, income protection and critical illness coverPrivate medical insurance and dental careEmployee Assistance ProgramCompetitive group pension planCyclescheme, Techscheme and season ticket loansVarious perks such as free Wednesday lunch in-office, on-site massages and regular social eventsLearning and development opportunities including in-house training and coaching, professional certifications, and coursesIf otherwise eligible, participation in the discretionary annual bonus programIf otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program #J-18808-Ljbffr