Data Engineer

Role: Data Engineer

Location: Sheffield (Hybrid - 3 days a week)

Employment: Fulltime

About the Role:

• We are looking for an experienced Azure Data Engineer to support the design, development and delivery of modern data platform solutions, with a particular focus on Microsoft Fabric migration and implementation.

• You will play a key role in delivering data engineering and migration activities, working closely with engineering teams and stakeholders to deliver high-quality, scalable and well-tested data solutions.

• The role requires strong technical expertise across Azure data technologies, alongside the ability to provide technical guidance, communicate effectively and contribute to successful delivery.

Key Responsibilities:

• Configure data ingestion processes and solutions.

• Lead the execution of Microsoft Fabric migration activities.

• Lead and guide migration engineers and QA resources.

• Develop Python, PySpark and Spark SQL code for data validation and curation within notebooks.

• Create data integration test cases and support testing activities.

• Implement and amend data processing and worker pipelines.

• Implement data validation and curation rules.

• Translate data models and mappings into detailed technical data mappings.

• Design and implement data integrations.

• Support the implementation and execution of data migration activities.

• Perform data analysis and profiling.

• Configure Azure DevOps / GitHub environments and CI/CD processes for Azure Data Factory and Databricks code.

Essential Skills & Experience:

• We are looking for candidates with:

• Strong experience designing and delivering Azure-based data platform solutions.

• Experience with technologies including: o Microsoft Fabric o Power Apps o Azure Synapse o Azure Data Factory (ADF) o Databricks o Azure Functions o Azure App Service o Logic Apps

• o Azure Kubernetes Service (AKS) o Web Apps

• Hands-on expertise in PySpark, SQL and Python.

• Good knowledge of real-time streaming applications, preferably including Kafka, Azure Functions or Azure Service Bus.

• Strong understanding of Spark processing and performance tuning.

• Experience working with file formats and partitioning, including Parquet, JSON, XML and CSV.

• Experience with Azure DevOps / GitHub and CI/CD.

• Strong programming experience in Python, with knowledge of other programming languages beneficial.

• Experience with both synchronous and asynchronous interface approaches.

• Knowledge of data modelling, including 3NF, dimensional modelling and Data Vault 2.0.

• Experience working within Agile delivery environments.

• Ability to produce comprehensive technical documentation.

• Ability to set and manage realistic expectations around timescales, costs, benefits and measures of success.

• Power BI experience is desirable.

The role requires experience across the following skill areas:

• Data Engineering

• Software Design

• Analytics

• Data Management

• Data Modelling and Design

• Data Visualisation

• Database Design

• Release and Deployment

Nice to Have

• Experience integrating and implementing data cataloguing tools such as Azure Purview / Collibra.

• Experience implementing and integrating visualisation tools such as Power BI or Tableau.

• Experience with C# application development, including ASP.NET MVC.