Comment on:

The following comment refers to this/these guideline(s)

Guideline 7

Cross-phase quality assurance

Researchers carry out each step of the research process lege artis. When research findings are made publicly available (in the narrower sense of publication, but also in a broader sense through other communication channels), the quality assurance mechanisms used are always explained. This applies especially when new methods are developed.

Explanations:

Continuous quality assurance during the research process includes, in particular, compliance with subject-specific standards and established methods, processes such as equipment calibration, the collection, processing and analysis of research data, the selection and use of research software, software development and programming, and the keeping of laboratory notebooks.

If researchers have made their findings publicly available and subsequently become aware of inconsistencies or errors in them, they make the necessary corrections. If the inconsistencies or errors constitute grounds for retracting a publication, the researchers will promptly request the publisher, infrastructure provider, etc. to correct or retract the publication and make a corresponding announcement. The same applies if researchers are made aware of such inconsistencies or errors by third parties.

The origin of the data, organisms, materials and software used in the research process is disclosed and the reuse of data is clearly indicated; original sources are cited. The nature and the scope of research data generated during the research process are described. Research data are handled in accordance with the requirements of the relevant subject area. The source code of publicly available software must be persistent, citable and documented. Depending on the particular subject area, it is an essential part of quality assurance that results or findings can be replicated or confirmed by other researchers (for example with the aid of a detailed description of materials and methods).

Guideline 12

Documentation

Researchers document all information relevant to the production of a research result as clearly as is required by and is appropriate for the relevant subject area to allow the result to be reviewed and assessed. In general, this also includes documenting individual results that do not support the research hypothesis. The selection of results must be avoided. Where subject-specific recommendations exist for review and assessment, researchers create documentation in accordance with these guidelines. If the documentation does not satisfy these requirements, the constraints and the reasons for them are clearly explained. Documentation and research results must not be manipulated; they are protected as effectively as possible against manipulation.

Explanations:

An important basis for enabling replication is to make available the information necessary to understand the research (including the research data used or generated, the methodological, evaluation and analytical steps taken, and, if relevant, the development of the hypothesis), to ensure that citations are clear, and, as far as possible, to enable third parties to access this information. Where research software is being developed, the source code is documented.

Guideline 13

Providing public access to research results

As a rule, researchers make all results available as part of scientific/academic discourse. In specific cases, however, there may be reasons not to make results publicly available (in the narrower sense of publication, but also in a broader sense through other communication channels); this decision must not depend on third parties. Researchers decide autonomously – with due regard for the conventions of the relevant subject area – whether, how and where to disseminate their results. If it has been decided to make results available in the public domain, researchers describe them clearly and in full. Where possible and reasonable, this includes making the research data, materials and information on which the results are based, as well as the methods and software used, available and fully explaining the work processes. Software programmed by researchers themselves is made publicly available along with the source code. Researchers provide full and correct information about their own preliminary work and that of others.

Explanations:

In the interest of transparency and to enable research to be referred to and reused by others, whenever possible researchers make the research data and principal materials on which a publication is based available in recognised archives and repositories in accordance with the FAIR principles (Findable, Accessible, Interoperable, Reusable). Restrictions may apply to public availability in the case of patent applications. If self-developed research software is to be made available to third parties, an appropriate licence is provided.

In line with the principle of “quality over quantity”, researchers avoid splitting research into inappropriately small publications. They limit the repetition of content from publications of which they were (co-)authors to that which is necessary to enable the reader to understand the context. They cite results previously made publicly available unless, in exceptional cases, this is deemed unnecessary by the general conventions of the discipline.

Replication as a component of quality assurance in the humanities and social sciences

Systematic testing and critical questioning of findings is fundamental to the research process. Not just in the social sciences but in certain research approaches used in the humanities, too (e.g. in archaeology and the digital humanities), replication – alongside other procedures – is an important element when it comes to carrying out quality assurance on research findings. This is especially true if the results were obtained using quantitative methods (see DFG statement, 2017). Replication and replicability are themselves the subject of investigations into science theory and methodology. In order to be able to categorise the different facets of replication in the humanities and social sciences and the different uses of the term in the literature, it is helpful to draw a distinction between two dimensions (Freese & Peterson, 2017):

1) Is the original dataset being used for replication or will new data be collected?

2) Is the replication study very similar in methodology to the original study or are there clear differences?

This gives rise to four different forms of replication:

a) replicating a study using the original data set and the same analysis steps as in the original study;
b) replicating a study using the original data set but different analysis steps;
c) replicating a study using newly collected data and applying a methodology and analysis that are as similar as possible to the original study (often referred to as “direct replication”);
d) replicating a study using newly collected data but with differences in methodology (often referred to as “conceptual replication”).

In many areas of the humanities and social sciences, all forms of replication are considered important elements of quality assurance, especially those in categories b), c) and d). Like the investigation of “new” effects, they also contribute to the cumulative gain of knowledge and should therefore generally be published; some journals offer dedicated sections for this purpose. Conducting replication studies of type a) and b) requires the original authors to document and make available their research data in a way that is transparent to third parties (see also commentary “Handling research data in the humanities and social sciences”). In order to enable third parties to conduct a replication study according to type c) and d), a comprehensive description of the entire methodology must be provided in publications. This includes a detailed description of the hypotheses, the sample, the study procedure, the materials used, the software and the programme code, all variables collected and all statistical analyses carried out.

In the humanities and social sciences, many of the phenomena and effects studied may depend on the context (e.g. time and place of study, characteristics of the subjects). As such, the question of the conditions required for a replication to be able to be considered “direct” is of crucial importance.

With several replication studies indicating a low level of replicability of research findings, there has been intense academic debate for several years in some subject areas on the possible causes of this. This has also resulted in a number of recommendations to increase replicability. These concern sample planning, study design, statistical analysis and research data management, as well as hypothesis and theory building. In order to ensure quality assurance of research findings and a cumulative gain in knowledge, it is important for researchers to reflect on questions of the replicability and replication of findings when planning and carrying out their research project. Publishers and infrastructure institutions likewise have an important role to play here: they contribute significantly to facilitating the implementation and publication of replication studies. The link list contains references to some Leibniz Institutes working in the areas of the social and behavioural sciences that provide helpful resources and services for conducting replication studies.

The comment belongs to the following categories:

GL7 (Humanities and social sciences) , GL12 (Humanities and social sciences) , GL13 (Humanities and social sciences)

|