Reach for the (cloudy) Sky! Exploring cloud-to-cloud archiving of digital records

Abstract

The National Archives (UK) is committed to making transfer of digital records faster, easier and more scalable. Our Transfer Digital Records (TDR) service allows depositors to upload records from a network drive environment. However, an increasing number of users (c86% of UK public record bodies) now use cloud ecosystems, mostly SharePoint but also Google drive. Our current digital transfer process requires users to extract records from the cloud before transfer (through re-upload to the cloud). This is slow, inefficient, error-prone and, most importantly, can disrupt or lose metadata (particularly dates). It also introduces challenges in selecting export formats. A significant volume of records are affected – both current records and legacy data that has been migrated to the cloud. Scalability is essential. This lighting talk will describe a feasibility study into enabling users to send records directly from depositors’ SharePoint services to the Archive, quickly, securely and without loss of metadata or file integrity. The study engaged with external partners including Microsoft. Criteria for maintaining integrity were identified: • Transfer user is clearly identified throughout the process • Transfer consignments are identified and tracked • Users can select which records to transfer • The Archive can securely access only selected records • Comprehensive metadata is harvested without disruption • Metadata is in a form suitable for ingest into archival systems • Files are harvested and uploaded to the Archive for ingest • Harvesting triggers AWS step-functions which automatically initiate archival processing • User administration, registration and compliance policies are met • Security and accessibility standards are met The study identified implementation options including full or minimal integration with a 3rd party app and bespoke development. The talk will outline the options and discuss some outstanding questions which we continue to explore: • Research into usability and adoption rates • Research into how depositors organise SharePoint metadata • Understanding how the Archive wishes to consume and process harvested records and metadata, including custom metadata and tags • Scaling constraints on records size or volume • Sustainability of the app approach, including ownership models, costs, maintenance, support, exit strategy

Details

Creators

Kirsten Arnold

Institutions

Date

2024-09-19 13:40:00 +0100

Keywords

metadata standards and implementation; scaling up

Publication Type

lightning talk

License

UK Open Government Licence v3

Download

(unknown) bytes

Slides

here

Video Stream

here

Collaborative Notes

here

View This Publication