Senior Data Engineer - Python, Databricks & Counterparty Credit Risk

Senior Data Engineer - Python, Databricks & Counterparty Credit Risk

Location: London - Hybrid

We are recruiting for an experienced Senior Data Engineer to join a leading global banking organisation, working across a highly technical data environment supporting Counterparty Credit Risk (CCR), Credit Risk and wider risk data.

This is a hands-on engineering position requiring strong expertise across Python, PySpark, SQL and Databricks, alongside experience designing, building and optimising scalable data pipelines and Lakehouse solutions.

Candidates should ideally have previously worked with Counterparty Credit Risk or Credit Risk data, including areas such as counterparty/exposure data, risk analytics, regulatory risk or associated banking risk datasets.

Key Responsibilities

Design, develop and maintain scalable data pipelines using Python, PySpark and Databricks

Build and optimise ETL/ELT workflows using Spark and Delta Lake

Engineer and transform data supporting Counterparty Credit Risk, Credit Risk and wider risk functions

Work with datasets relating to areas such as counterparties, credit exposures, limits, collateral and risk analytics

Design robust data models and architectures across Lakehouse / Medallion environments

Develop and manage Databricks notebooks, jobs and workflows

Optimise large-scale distributed data processing and troubleshoot Spark performance

Implement strong data quality, reconciliation, governance and monitoring controls

Build reliable logging and alerting across production data pipelines

Integrate data platforms with Power BI dashboards and wider applications

Support CI/CD and engineering best practices across development, testing and production environments

Work closely with Risk, Technology, Analytics and other senior stakeholders

Required Experience

10+ years' experience across Data Engineering or closely related data disciplines

Strong hands-on Python development experience

Strong PySpark / Apache Spark engineering experience

Extensive hands-on experience with Databricks, including notebooks, jobs/workflows and Delta Lake

Advanced SQL and relational database experience

Strong track record designing and building scalable ETL/ELT data pipelines

Experience with Delta Lake, Lakehouse and/or Medallion architectures

Experience handling banking risk data, ideally within Counterparty Credit Risk (CCR), Credit Risk, counterparty exposure or closely related risk domains

Strong understanding of data modelling, data quality and production data engineering

Cloud experience across Azure, AWS and/or GCP

Experience with modern data platforms/warehouses such as Snowflake, BigQuery or Redshift

Experience integrating data platforms with Power BI

Experience with orchestration such as Databricks Workflows or Airflow

Highly Desirable

Direct experience engineering data for Counterparty Credit Risk, including exposure/counterparty datasets

Knowledge of areas such as PFE, EPE, EAD, CVA/XVA, SA-CCR, collateral, netting or counterparty limits

Previous experience within an investment bank, global bank or capital-markets environment

Databricks Data Engineer certification

Kafka / Structured Streaming experience

Delta Lake CDC, Time Travel and optimisation

CI/CD, Git and DevOps experience

Docker / Kubernetes experience

Experience exposing data through APIs or supporting React-based front-end applications

Experience supporting analytics or ML workloads within Databricks

We are particularly interested in genuinely hands-on Senior Data Engineers who can demonstrate the Databricks and Python solutions they have personally designed, built, optimised and supported in production.

#J-18808-Ljbffr