DaQS - Developing a scientific service for the reuse of questionnaire items in the context of educational research

The ‘Database for Quality of Schools – DaQS’ is a special service for educational researchers, offered in the context of the German ‘Research Data Centre Education’. Currently the service is only available in German language but nevertheless all researchers are invited to search, enquire, browse and explore contents and background information on several documented studies and the used research instruments. Following the idea of holistic open science – including principles such as open access availability of research outcomes like publications, research instruments and data, transparency, reproducibility, confirmability or reassessment – the disciplinary service ‘DaQS’ presents an approach to dealing with (background) questionnaires as a special type of (research) resource. On the basis of a central repository, enduring availability is facilitated and re-use by the educational research community is supported.
Based on this offer, researchers are able to combine existing questionnaire instruments resp. items to create a new one according to their specific research questions. The implemented documentation of statistical values allows comparison of several instruments of the same content for use in a new questionnaire. The general goal is to build up and maintain an infrastructure for a sustainable storage of questionnaire items in a central open access repository as an approach to support questionnaire development. This implies a guarantee of a reliable archiving of item collections at a permanently existing location. Furthermore, comfortable and intelligent search resp. retrieval options are provided that facilitate finding and comparison of a set of items. Another issue refers to maintaining a chronological overview of item development from a historical perspective, i.e. the origin of an item is also documented. Other purposes focus on the transparency of the research process in general with regard to reproducibility and improvement of the overall quality of questionnaire development by offering validated questions resp. single items.
In terms of information science, DaQS was especially committed to generating an adequate set of relevant metadata with regard to specific requirements in the domain of educational research. This instrument will be applied to enhancing standardization and harmonization in questionnaire development. In this context, it is necessary to set up a general structure that serves for different research instruments from various studies. Therefore, a user-centred design approach was pursued - including initial experts’ interviews and continuous user-centred system validation.
Presently the database comprehends a total of approx. 16,000 items originating from several large and small scale studies. The manner of content storage will be displayed on the basis of the developed metadata scheme, which is derived from the standard DDI (data documentation initiative). Essential elements are the items on a literal basis, answer formats, information about the target group, citation of origin and item statistics giving evidence about the item quality (following classical test theory and item response theory). This opens the system for interoperability with other related databases and systems. Besides sophisticated search options which have been set up on the internal structure, the repository features another individual solution to accessing the content by browsing. This is based on a theoretical approach and offers subject-related access via thematic categories, which build up a framework for opening up the item collection. The framework refers to an elementary input-process-output model dealing with the dimensions of input (individual and institutional background), processes (on school and classroom level), and output (effects on students’ competences). All instruments are dedicated to these theoretical categories of educational research upon two lower levels of constructs. This three-level categories scheme builds the theoretical backbone of the database. It was created by involving several meta-analyses (e.g. Scheerens & Bosker 1997, Ditton 2000, Seidel & Shavelson 2007, Seidel 2008) and the evaluation of the available instruments at the beginning of the project. Now the categories-scheme is still increasing by adding new studies with their specific aims and issues. This approach leads to an inductive categorisation, creating new classes in relation to the documented data. Following the idea of participation in web 2.0 a special feature invites users to validate the categories-scheme by making their own suggestions to categorise the instruments.
In summary, the item bank is a centralized archive for questionnaires on a national level which provides the opportunity for a structured documentation of sets of questions and items. Although the repository is open to everyone, the target audience is educational research and academic teaching. Technically, the system is based on a web-based service embedding a relational database. The service has implemented the strategy to facilitate citation and re-use of content by providing “doi” (digital object identifier) as part of the corresponding metadata.
To answer users’ needs for easy availability of data, research instruments or publications via a single point of access, further steps will be taken to document the entire context of studies. An integrated approach is envisaged in terms of a ‘Research Data Centre Education’, which will offer describing information on a study on the one hand and study outcomes including research instruments, research data, and publications on the other hand. Furthermore, the service will be accommodated to the conditions of increasingly emerging mixed method studies (quantitative, qualitative research).
Currently development work is done to enhance the item bank with additional components which are further steps towards the aim of documenting research studies on a global level. Respectively, documentation of assessment tests (first paper-based only) used in educational research will be documented. This step will definitely enrich the instruments especially on the output level of the categories-scheme. Another component will offer the possibility of long-term storage of qualitative research data – this includes video and audio data as well as transcript of records. Although this future service has to deal with restrictions according to sensibility of personalized data – in contrast to the existing open access repository – it can be illustrated why it is still relevant to the scientific community.
Last but not least, future development will cover the design of tools which facilitate the use of the documented instruments. A recent activity concerns the design of a software for questionnaire building (known as ‘Qbee’), which allows to adopt questions from the central item bank into an individual process of questionnaire development. This enables researchers from different locations to collaborate with the option to automatically feed back the resulting questionnaire into the item database ‘DaQS’.