Lead Data Engineer

Location: Greater Manchester (Hybrid, 1 - 2 days a week on-site)

Salary: £80,000 - £85,000 (potentially some wiggle room for the right person)

We're working with an established fintech business in Greater Manchester that is doing something genuinely interesting with AI right now. Not pilots. Not proof of concepts gathering dust. Real, production-grade AI embedded into customer journeys and internal operations, and they're only just getting started.

They've recently brought in a Head of Data and AI who has both the vision and the organisational backing to push this forward hard. This role sits right at the centre of that push, owning the data platform that everything else gets built on.

The Role

This is a hands-on Lead Data Engineer role reporting into the Head of Data & AI. You'll take on the existing data engineering estate and bring state-of-the-art batch and live data processing to it, so it can properly serve BI, Data Science and Agentic AI use cases going forward.

It's a lead role with line management, but a genuinely hands-on one. You'll be expected to use tools like Claude Code as a core part of how you engineer and build, push the technology envelope where it's warranted, and bring a small technical team with you as you do it.

The aim isn't to rebuild for the sake of it. It's to evolve what works, automate what shouldn't be manual, and drive towards one platform that serves the whole business rather than a fragmented, function-based estate.

What You'll Be Doing

• Improving and evolving the existing data estate so it's ready for modern batch and live-streaming data processing
• Simplifying and automating data processes that are manual, duplicated or fragile today
• Supporting the BI, Data Science and Agentic (Applied AI) teams from a single platform
• Building a semantic layer over the data and exposing it through MCP and similar interfaces, giving the business the ability to analyse its data through tools like Claude
• Delivering a governed self-serve capability that lets anyone in the business query data safely
• Leading the design of a Customer Data Platform for batch and live data processing, working closely with ML and Data Science on shared ingestion, storage and feature needs
• Line-managing the data engineers, including hiring and developing the team needed to deliver
• Owning the governance, security and running cost of the data platform alongside Data & AI leadership

What You'll Need

• Proven delivery in both batch processing (warehouse loads, ELT, orchestration) and modern streaming, with Flink or comparable engines
• Production AWS data engineering experience: S3, Lambda, Glue, Athena/Iceberg, Kinesis or equivalent streaming services, RDS, IAM, CloudWatch, plus infrastructure as code with Terraform and CI/CD
• Strong lakehouse and data modelling depth: Iceberg-style open table formats, medallion-style layering, dimensional modelling, and the judgement to know when a warehouse should consume from a lakehouse rather than replicate it
• Strong SQL and Python with proper testing and engineering discipline. Willingness to work with SQL Server, SSIS/SSDT and Power BI at the start, even if you've moved beyond them elsewhere
• Experience designing semantic layers, conformed metric definitions and glossaries, and exposing data to LLM tooling through MCP servers or similar
• Daily hands-on use of Claude Code or an equivalent frontier coding assistant as part of how you engineer, with a clear story of how it's made you materially faster without compromising quality or security
• Solid data governance experience in a regulated business: retention, lineage, classification, PII minimisation, auditability and PCI-DSS boundaries built into the platform, not bolted on
• Business-area fluency, learning what functions like FinCrime, Regulatory Reporting, Product, CX, Finance and Marketing actually do with data, and prioritising from their outcomes
• Experience line-managing a small technical team in a delivery setting, including hiring and development
• Clear written and spoken communication with senior stakeholders, including architecture position papers and candid risk statements

Bonus Points For

• Experience in financial services or another regulated environment
• PySpark for large-scale batch or streaming transformations
• Migrating a SQL Server/SSIS estate to a metadata-driven or lakehouse architecture, and a clear view on what you'd do differently
• Designing or operating a Customer Data Platform or event platform (Kinesis, Flink, StarRocks or comparable)
• Building data-quality and data-contract tooling (dbt tests, Great Expectations or equivalent) and synthetic monitoring around feeds
• Authoring Claude skills or MCP servers for a team, or running an LLM natural-language layer over BI data
• FinCrime, fraud or regulatory-reporting data domains
• Power BI at the platform level (semantic models, gateway, administration) for the transition period