What is Reproducible Research?

Estimated time: 10 minutes

For our workshop today….

---

For our workshop today, as the training was designed to be relevant to all disciplines, we are using the UKRN’s broad definition of reproducibility:

“Research that is sufficiently transparent that someone with the relevant expertise can clearly follow, as relevant for different types of research:

• how it was done;

• why it was done in that way;

• the evidence that it established;

• the reasoning and/or judgements that were used; and

• how all of that led justifiably to the research findings and conclusions.”

In most discussions of reproducible research, especially in STEM disciplines, reproducible research is the concept that someone can, given your data and methods, redo your research and come to the same conclusion.

We expect that anyone conducting the same research should always come to the same conclusion. This enables us to trust that the knowledge derived from this research can be accepted as fact.

Replicability and Repeatability

---

There are a number of terms you may hear when we talk about reproducible. Replicable and Repeatable are two common terms that may arise.

The Turing Way offers the following descriptions:

Reproducible

Replicable

Robust

Generalisable

A result is reproducible when the same analysis steps performed on the same dataset consistently produces the same answer.

A result is replicable when the same analysis performed on different datasets produces qualitatively similar answers.

A result is robust when the same dataset is subjected to different analysis workflows to answer the same research question (for example one pipeline written in R and another written in Python) and a qualitatively similar or identical answer is produced. Robust results show that the work is not dependent on the specificities of the programming language chosen to perform the analysis.

Combining replicable and robust findings allow us to form generalisable results. Note that running an analysis on a different software implementation and with a different dataset does not provide generalised results. There will be many more steps to know how well the work applies to all the different aspects of the research question. Generalisation is an important step towards understanding that the result is not dependent on a particular dataset nor a particular version of the analysis pipeline.

Let’s look now on why this is important.

References

Reference: Definitions — The Turing Way Community. (2022). The Turing Way: A handbook for reproducible, ethical and collaborative research (Version 1.1.0) [Computer software]. https://doi.org/10.5281/zenodo.3233853 licenced as CC-BY

The Turing Way Community. This illustration is created by Scriberia with The Turing Way community, used under a CC-BY 4.0 licence. DOI: https://doi.org/10.5281/zenodo.3332807

Key Points

In this lesson, we have learnt:

• What is reproducible research?

• The different terms around reproducibility