Data Engineer - #2165918

Location: Remote working
Type: Contract or Permanent
Day Rate: Up to £550 per day Inside IR35
Salary: Up to £90,000 package
Security Clearance: SC Eligible (Mandatory) / Active SC Preferred
Start Date: Immediate or up to 2 weeks' notice

We are supporting a major programme seeking an experienced Data Engineer with strong Databricks and Geospatial Data expertise. You will play a key role in designing and delivering scalable data pipelines that combine demographic datasets with geographic reference data to support population analysis, reporting, service planning, and operational decision-making.

This is an excellent opportunity for a Data Engineer who enjoys working with large-scale data platforms, geospatial processing, and modern cloud-based technologies in a highly collaborative environment.

• Design, build and support data pipelines using Databricks, Python, PySpark and SQL
• Develop reusable ingestion, transformation, validation and enrichment processes
• Ingest demographic datasets, postcode data, geographic boundaries and reference datasets
• Build and maintain Delta Lake datasets and manage data lineage
• Perform postcode enrichment and geographic assignment processing
• Implement spatial joins, point-in-polygon matching and location-based calculations
• Manage geographic reference data versions and historical reproducibility
• Develop automated data quality controls and reconciliation processes
• Optimise Spark workloads and pipeline performance
• Support production releases, monitoring and troubleshooting activities
• Work closely with analysts, data owners and platform teams to define business rules and acceptance criteria

• Strong commercial experience with:
  • Databricks
  • Python
  • PySpark
  • SQL
• Proven experience building and supporting production-grade data pipelines
• Strong knowledge of Delta Lake, schema management, merge logic and incremental processing
• Experience handling large datasets within cloud-based environments
• Practical experience working with geospatial data and spatial processing
• Understanding of coordinate reference systems, including:
  • British National Grid (BNG)
  • WGS84
• Experience with spatial joins, geometry validation and geographic aggregation
• Strong data quality, reconciliation and testing experience
• Experience with Git, CI/CD and automated deployment pipelines

• Apache Sedona or Databricks Spatial Functions
• GeoPandas, Shapely, pyproj or QGIS
• NHS or healthcare data
• NHS Postcode Directory (NHSPD)
• ONS Postcode Directory (ONSPD)
• National Statistics Postcode Lookup (NSPL)
• Experience working with:
  • LSOAs
  • MSOAs
  • Local Authorities
  • Integrated Care Boards (ICBs)
• GeoJSON, GeoParquet, Shapefiles, WKT/WKB formats
• Population, census or demographic datasets
• Experience working within regulated or secure environments

The ideal candidate will have a strong Data Engineering background combined with experience working with geographic and location-based datasets. You will be comfortable building scalable Databricks solutions, implementing automated processes, and ensuring the accuracy and reliability of demographic and geospatial data.

Candidates must be eligible to obtain SC Clearance, with active clearance highly advantageous.

• Up to £550 per day
• Inside IR35
• Predominantly remote working

• Up to £90,000 total package
• Base salary up to approximately £81,000
• Performance bonus
• Predominantly remote working

Due to the urgency of this requirement, we are particularly interested in candidates who are immediately available or on a notice period of no more than two weeks.

Apply today with your latest CV for immediate consideration.