Senior Data Engineer

Clearance required - active DV clearance

Client sites in Wiltshire full time. Potentially other UK site travel as well.

Opportunity to condense the hours of work.

This role is inside IR35.

What you will do as a Senior Data Engineer

Design, build and maintain secure, scalable data solutions across the full lifecycle, including acquisition, integration, storage, processing and analysis.

Develop and maintain data ingestion and transformation pipelines using Apache NiFi, Python and SQL.

Design data models, schemas and data contracts / Interface Control Documents (ICDs) to support reliable integration between systems.

Integrate data from REST APIs and other source systems, working with Parquet and S3-compatible object storage.

Implement data validation, assurance, metadata management and traceability to ensure data is accurate, complete and fit for purpose.

Develop tools for data-driven insights and task automation, and research and adopt appropriate emerging technologies.

Collaborate with customer and internal stakeholders to define data, interface and tool requirements.

Work closely with DevOps Engineers, Data Scientists, Analysts and Cyber Consultants to deliver integrated solutions.

Take ownership of deliverables and ensure the quality, security and maintainability of outputs.

What you'll bring

Proven experience in data engineering, with the ability to work independently and take ownership of customer-facing deliverables.

Strong Python and SQL skills, with experience building, testing and optimising data pipelines.

Hands-on experience with Apache NiFi for data ingestion, routing, transformation and orchestration.

Experience of data modelling and schema design, including defining data contracts or Interface Control Documents (ICDs).

Experience working with Parquet and S3-compatible object storage.

Experience integrating REST APIs and troubleshooting data exchanges between systems.

Experience deploying or supporting data solutions within OpenShift or Kubernetes environments.

Experience using GitLab CI/CD to automate build, test and deployment activities.

Understanding of data validation and assurance approaches, including quality controls and reconciliation.

Understanding of metadata management, data lineage and traceability.

Strong problem-solving skills and the ability to collaborate across m...