An evaluation of the building of a new tool to test the scalability of a trusted digital repository

One of the aims of the Digital Repository of Ireland project is to construct a trusted digital humanities repository. A key requirement for a repository to qualify as trusted is that the users must be able to rely on it to remain functioning as expected, even under very high data loads. Thus, the challenge of testing the scalability of prototype solutions with large-scale datasets arose. As the DRI project is at an early stage of development, assigned datasets have yet to be repurposed for ingestion into DRI. Five test projects had been identified, named ‘demo projects’, which would come on board during the DRI project timeline, but these projects are coming on board in staggered stages. Furthermore, the metadata attached to these demo projects were either not created yet or not in suitable formats for use in the DRI. One project used a custom database schema that would not meet international metadata standards.

Thus, a solution of building a tool called Hydrate, that would generate large sets of dummy data and metadata was proposed. These large-scale datasets would be ingestible into the digital repository prototypes for scalability testing of these prototypes. The proposed tool would be written in Rails, in order to sit within the Hydra framework already agreed upon for use in the DRI project. Java was also examined as a potential implementation language, but discarded in favour of Rails in order to ensure greater integration with the DRI project as a whole, as well as a higher likelihood of future use in other Hydra-based projects. This solution would also allow for the requirements already identified for the DRI to be tracked and tested properly in metadata generation.