From days to minutes: How NHS England Secure Data Environment is accelerating healthcare research – NHS England Digital

Once a research project has a Data Sharing Agreement, a secure workspace within the Secure Data Environment (SDE) is created for the approved researcher(s) to access the selected datasets. All data provisioned into the SDE platform must be validated. This process was manual and took on average 3 days to complete.

I, along with Helen Richardson and Elizabeth Kelly in the Service team decided to make significant improvements in this area. We focused on:

• improving the efficiency and consistency of the data validation process
• making it a reusable process to save time and follow best practice
• sharing the code so others can benefit.

Using Python, we created reusable programming code that thoroughly checks for problems with the data tables. We also engaged with stakeholders to ensure validation checks were comprehensive.

Through these innovations, we reduced the time to validate data from 3 days to an astonishing 30 minutes. This dramatic reduction in validation time allows researchers to begin their work much more quickly, significantly accelerating the pace of healthcare research.

The validation code is now reusable across different datasets, promoting efficiency and best practices. By enabling earlier identification and resolution of potential issues, this new approach not only saves time but also enhances the quality and reliability of healthcare data available for research.

One of the themes at HACA2024 was that good analysis saves lives; and good analysis is built on good quality data. By ensuring our data is validated in a consistent, reliable and transparent way we lay the groundwork for future analysis. Saving time as well is the icing on the cake. Some other attendees had mentioned they had been doing similar efforts in their own trusts and organisations; by sharing this work we can facilitate that and help everyone improve.