Senior Data Engineer - Python, Databricks & Counterparty Credit Risk - Venn Group - #2157673

Senior Data Engineer – Python, Databricks & Counterparty Credit Risk

Location: London

Working Pattern: Hybrid – 2–3 days per week onsite

We are recruiting for an experienced Senior Data Engineer to join a leading global banking organisation, working across a highly technical data environment supporting Counterparty Credit Risk (CCR), Credit Risk and wider risk data .

This is a hands-on engineering position requiring strong expertise across Python, PySpark, SQL and Databricks , alongside experience designing, building and optimising scalable data pipelines and Lakehouse solutions.

Candidates should ideally have previously worked with Counterparty Credit Risk or Credit Risk data , including areas such as counterparty/exposure data, risk analytics, regulatory risk or associated banking risk datasets.

Key Responsibilities

1. Design, develop and maintain scalable data pipelines using Python, PySpark and Databricks
2. Build and optimise ETL/ELT workflows using Spark and Delta Lake
3. Engineer and transform data supporting Counterparty Credit Risk, Credit Risk and wider risk functions
4. Work with datasets relating to areas such as counterparties, credit exposures, limits, collateral and risk analytics
5. Design robust data models and architectures across Lakehouse / Medallion environments
6. Develop and manage Databricks notebooks, jobs and workflows
7. Optimise large-scale distributed data processing and troubleshoot Spark performance
8. Implement strong data quality, reconciliation, governance and monitoring controls
9. Build reliable logging and alerting across production data pipelines
10. Integrate data platforms with Power BI dashboards and wider applications
11. Support CI/CD and engineering best practices across development, testing and production environments
12. Work closely with Risk, Technology, Analytics and other senior stakeholders

Required Experience

1. 10+ years' experience across Data Engineering or closely related data disciplines
2. Strong hands-on Python development experience
3. Strong PySpark / Apache Spark engineering experience
4. Extensive hands-on experience with Databricks , including notebooks, jobs/workflows and Delta Lake
5. Advanced SQL and relational database experience
6. Strong track record designing and building scalable ETL/ELT data pipelines
7. Experience with Delta Lake, Lakehouse and/or Medallion architectures
8. Experience handling banking risk data , ideally within Counterparty Credit Risk (CCR), Credit Risk, counterparty exposure or closely related risk domains
9. Strong understanding of data modelling, data quality and production data engineering
10. Cloud experience across Azure, AWS and/or GCP
11. Experience with modern data platforms/warehouses such as Snowflake, BigQuery or Redshift
12. Experience integrating data platforms with Power BI
13. Experience with orchestration such as Databricks Workflows or Airflow

Highly Desirable

1. Direct experience engineering data for Counterparty Credit Risk , including exposure/counterparty datasets
2. Knowledge of areas such as PFE, EPE, EAD, CVA/XVA, SA-CCR, collateral, netting or counterparty limits
3. Previous experience within an investment bank, global bank or capital-markets environment
4. Databricks Data Engineer certification
5. Kafka / Structured Streaming experience
6. Delta Lake CDC, Time Travel and optimisation
7. CI/CD, Git and DevOps experience
8. Docker / Kubernetes experience
9. Experience exposing data through APIs or supporting React-based front-end applications
10. Experience supporting analytics or ML workloads within Databricks

We are particularly interested in genuinely hands-on Senior Data Engineers who can demonstrate the Databricks and Python solutions they have personally designed, built, optimised and supported in production.

If you have strong Databricks, Python and PySpark engineering experience alongside exposure to Counterparty Credit Risk, Credit Risk or banking risk data , please apply for further information.