Research Activities and Uncertainty
Scientific studies impact our daily lives with national and international effects. Therefore, the results of such studies have great importance. However, an issue that is concerning is whether these scientific studies can be repeated with the same results.
The National Research Council ran a consensus study (NASEM 2019) that addresses the repeatability of scientific studies by defining primary ideas, the presence of problems that can affect the results of the studies, and how to deal with them. They (NASEM) state that the repeatability of a scientific study, along with the reproduction of identical results should be considered the same problem, despite what others think.
A strategic approach to research is necessary, that is focused on the gathering of evidence identical in all similar studies. By gathering evidence in the same way, the scientist can consider the issues that affect repeatability and other problems. By doing so, results that are repeated can be compared to "clean data" and opposing theories/ideas. The evidence gathered based on what to expect can proceed.
In many sciences, single studies cannot increase our knowledge of the subject itself. We cannot view individual study results as building blocks for that science. The evidence gathered requires multiple studies of the same experiment.
The comparison of results with expected predictions are more useful to science than just comparing the results of different studies. By comparing results to expected predictions, it only reinforces whether those predictions and theories/ideas are correct. Especially when comparing results to many theories/ideas which would allow confidence in those very same results and theories.
A systematic approach to reproducing similar results is to create an "evolving information state" or EIS, which is based on many ideas and their model examples. During the experiment, at any point in time, a model can carry weight in whether the results gathered create a degree of confidence in the data. So long as the data fits into the model, the weight of that data only reinforces confidence. Any new study of data, according to the time demonstrated in the model, would demonstrate a "weight of high confidence" of that data. A weight of 1 would demonstrate high confidence while a weight of 0 would show low confidence.
This model was used in a US study of Mallard Ducks, their breeding season, and the hunting regulations for the upcoming season. Two studies displayed a weight of 0 and required further manipulation of data, and the estimation of parameters/covariant, etc.
NASEM 2019 defines reproducibility as obtaining consistent results using the same data, computational steps, methods, codes and conditions. With a Multiple Study Approach to the accumulation of data, to enforce reproducibility would require checking type consistency of the results. When performed, such discrepancies in the data can be found. However, what to do with these discrepancies and how they affect future studies is not known. Different approaches to the gathering of data should affect the variations and their source, which would allow useful interpretation of the different data.
Data is used to reinforce theories created in science. Consistent data reinforces the confidence in these theories. Repeating the study provides more evidence that the theory and what it predicts, is correct. Preregistration of experiments is one approach for improving the ability to reproduce similar data. In such preregistration, the idea/theory and how it is measured are specified before beginning the study. This allows for the result to be considered exploratory or confirmatory (not knowing the results versus knowing what to expect). EIS believes it should go one step further by testing the idea/theory many times and progressing the study each time to accumulate more detailed evidence.
When considering Multiple Studies, EIS describes the research activities as identifying, retrieving, evaluating, synthesizing, interpreting, and contextualizing the data for systematic reviews and multiple-perspective analysis. The shift to prospective approach eliminates many problems with variations in data. EIS believes rather than using pre-experimental beliefs, confidence in data can be based on previous predictability. Does the data coincide with what we believe will happen?
NASEM 2019 recommends that researchers get away from isolated studies and instead contribute to programs that progressively accumulate data. Under one progressive theory, a consortium of research theory can focus on specific questions.
Finally, NASEM 2019 states that anyone making personal/public policy decisions based on evidence gathered should reconsider their decision. They instead should see a "Bigger Picture" using algorithms and methods developed to deal with uncertainty.
