Data Engineer

Data Engineer

• Contract duration: 6 months
• Location: Mostly remote, with travel to either London or Chesterfield once per month
• Pay range: £600.00- £650.00 per day/ Inside IR35
• Start date: ASAP

The Data Engineer will join our client’s new Data Enablement Programme, which is bringing data held across different parts of the organisation into a centralised Databricks platform.

Key Responsibilities include:

• Design, build, test and maintain production-ready batch and streaming data pipelines that provide trusted, timely and accessible data across the Enterprise Data Platform.
• Develop scalable, cloud-native data solutions using AWS services and Databricks, including workflows, notebooks and production jobs.
• Create robust data ingestion, transformation, orchestration and automation components that meet agreed functional, performance and operational requirements.
• Apply engineering, security and change standards throughout delivery, including peer review, documentation, source control and agreed methodologies.
• Implement unit, integration and regression testing to assure the quality, reliability and maintainability of delivered solutions.
• Use CI/CD pipelines to automate deployment and release processes, reducing manual intervention and improving delivery consistency.
• Collaborate with architects, analysts, product owners and business stakeholders to translate requirements into effective technical solutions.
• Contribute actively to Agile ceremonies, sprint commitments, platform milestones and roadmap objectives.
• Produce clear technical designs and documentation and provide effective knowledge transfer to support ongoing operation and maintenance.
• Identify and deliver improvements to platform scalability, performance, resilience and operational efficiency.
• Skills and Experience Needed
• Strong commercial experience in data engineering, including the design and delivery of production-grade batch and streaming pipelines.
• Hands-on experience with Databricks, including notebooks, workflows and jobs deployed to production.
• Practical experience using AWS data services, particularly S3, Glue and Lambda, together with related cloud technologies.
• Proficiency in data ingestion, transformation, orchestration and pipeline automation.
• Experience implementing CI/CD, source control, peer review and automated deployment practices.
• Strong software engineering discipline, including unit, integration and regression testing, documentation and maintainable code.
• Understanding of cloud security, engineering standards, operational support readiness and resilient solution design.
• Experience working in Agile, multidisciplinary delivery teams and managing work against sprint commitments.
• Ability to communicate technical concepts clearly and work effectively with architects, analysts, product owners and business stakeholders.
• A collaborative, delivery-focused approach, with the ability to identify improvements and contribute to platform modernisation.

Skills/ experience required include:

• SQL: strong practical experience querying, transforming and preparing data for reporting.
• Python: confident coding skills for data engineering and pipeline development.
• Data engineering: previous hands-on experience building pipelines that ingest data, store it in databases or centralised platforms, and expose it for reporting.
• Databricks: experience working with Databricks as a centralised data platform.
• AWS: practical experience with AWS knowledge of specific services such as S3, Glue and Lambda is useful but not mandatory.
• Software engineering: ability to write maintainable production-quality code, ideally gained through a software engineering background.
• CI/CD: familiarity with standard CI/CD pipelines, development workflows and industry engineering practices.
• Independent delivery: able to own work and progress tasks with limited supervision when other developers are supporting priorities elsewhere.
• Experience level: approximately five years’ relevant experience, with prior data engineering experience strongly preferred.
• Desirable Skills
• - Golang: anticipated as part of the technology stack, alongside Python and SQL.
• - Experience consolidating data from multiple source systems into a central platform.
• - Experience producing datasets that support management information, operational reporting and business performance analysis.