The replication crisis, also known as the reproducibility or replicability crisis, refers to widespread failures to reproduce published scientific results. Because the reproducibility of empirical results is the cornerstone of the scientific method, such failures undermine the credibility of theories and challenge substantial parts of scientific knowledge. Psychology and medicine have been focal points for replication efforts, with researchers systematically reexamining classic studies to verify their reliability and, when failures emerge, to identify the underlying causes. Data strongly indicates that other natural and social sciences are also affected. The phrase "replication crisis" was coined in the early 2010s as part of a growing awareness of the problem. Considerations of causes and remedies have given rise to a new scientific discipline known as metascience, which uses methods of empirical research to examine empirical research practice. Researchers distinguish two forms of reproducibility. Reproducibility in a narrow sense refers to reexamining and validating the analysis of a set of data. The second category, replication, involves repeating an experiment or study with new, independent data to verify the original conclusions.
Background
Replication Replication has been called "the cornerstone of science". Environmental health scientist Stefan Schmidt began a 2009 review with this description of replication:
Replication is one of the central issues in any empirical science. To confirm results or hypotheses by a repetition procedure is at the basis of any scientific conception. A replication experiment to demonstrate that the same findings can be obtained in any other place by any other researcher is conceived as an operationalization of objectivity. It is the proof that the experiment reflects knowledge that can be separated from the specific circumstances (such as time, place, or persons) under which it was gained. But no universal definition of replication or related concepts has been agreed on. Replication types include:
direct (repeating procedures as closely as possible), systematic (repeating with intentional changes), and conceptual (testing hypotheses using different procedures to assess generalizability). Reproducibility can also be distinguished from replication, as referring to reproducing the same results using the same data set. Reproducibility of this type is why many researchers make their data available to others for testing. Replication failures do not indicate that affected fields lack scientific rigor. Rather, they reflect the normal operation of science—a mechanism by which unsupported hypotheses are eliminated, but which often functions slowly and inconsistently. A hypothesis is generally considered supported when the results match the predicted pattern and that pattern is found to be statistically significant. Under null hypothesis assumption, results are deemed statistically significant when their probability falls below a predetermined threshold (the significance level). This generally answers the question of how unlikely such results would be by chance alone if no true effect existed in the statistical population. If the probability associated with the test statistic exceeds the chosen critical value, the results are considered statistically significant. The p-value represents the probability of obtaining results at least as extreme as observed, assuming the null hypothesis is true. The standard threshold p < 0.05 means accepting a 5% false positive rate. Some fields use smaller p-values, such as p < 0.01 (1% chance of a false positive) or p < 0.001 (0.1% chance of a false positive). But a smaller chance of a false positive often requires greater sample sizes or a greater chance of a false negative (a correct hypothesis being erroneously found incorrect). Although p-value testing is the most commonly used method, it is not the only one.
Statistics Certain terms commonly used in discussions of the replication crisis have technically precise meanings, which are presented here. In the most common case, null hypothesis testing, there are two hypotheses, a null hypothesis H 0 {\displaystyle H_{0}} and an alternative hypothesis H 1 {\displaystyle H_{1}} . The null hypothesis is typically of the form "X and Y are statistically independent". For example, the null hypothesis might be "taking drug X does not change 1-year recovery rate from disease Y", and the alternative hypothesis is that it does change. As testing for full statistical independence is difficult, the full null hypothesis is often reduced to a simplified null hypothesis "the effect size is 0", where "effect size" is a real number that is 0 if the full null hypothesis is true, and the larger the effect size is, the more the null hypothesis is false. For example, if X is binary, then the effect size might be defined as the change in the expectation of Y upon a change of X: ( effect size ) = E [ Y | X = 1 ] − E [ Y | X = 0 ] {\displaystyle ({\text{effect size}})=\mathbb {E} [Y|X=1]-\mathbb {E} [Y|X=0]} Note that the effect size as defined above might be zero even if X and Y are not independent, such as when their relationship is non-linear (such as Y ∼ N ( 0 , 1 + X ) {\displaystyle Y\sim {\mathcal {N}}(0,1+X)} ) or when one variable affects different subgroups oppositely. Since different definitions of "effect size" capture different ways for X and Y to be dependent, there are many definitions of effect size. In practice, effect sizes cannot be directly observed, but must be measured by statistical estimators. For example, the above definition of effect size is often measured by Cohen's d estimator. The same effect size might have multiple estimators, as they have tradeoffs between efficiency, bias, variance, etc. This further increases the number of possible statistical quantities that can be computed on a single dataset. When an estimator for an effect size is used for statistical testing, it is called a test statistic.
A null hypothesis test is a decision procedure which takes in some data, and outputs either H 0 {\displaystyle H_{0}} or H 1 {\displaystyle H_{1}} . If it outputs H 1 {\displaystyle H_{1}} , it is usually stated as "there is a statistically significant effect" or "the null hypothesis is rejected". Often, the statistical test is a (one-sided) threshold test, which is structured as follows:
Gather data D {\displaystyle D} . Compute a test statistic t [ D ] {\displaystyle t[D]} for the data. Compare the test statistic against a critical value/threshold t threshold {\displaystyle t_{\text{threshold}}} . If t [ D ] > t threshold {\displaystyle t[D]>t_{\text{threshold}}} , then output H 1 {\displaystyle H_{1}} , else, output H 0 {\displaystyle H_{0}} . A two-sided threshold test is similar, but with two thresholds, such that it outputs H 1 {\displaystyle H_{1}} if either t [ D ] < t threshold − {\displaystyle t[D]<t_{\text{threshold}}^{-}} or t [ D ] > t threshold + {\displaystyle t[D]>t_{\text{threshold}}^{+}}
There are 4 possible outcomes of a null hypothesis test: false negative, true negative, false positive, true positive. A false negative means that H 0 {\displaystyle H_{0}} is true, but the test outcome is H 1 {\displaystyle H_{1}} ; a true negative means that H 0 {\displaystyle H_{0}} is true, and the test outcome is H 0 {\displaystyle H_{0}} , etc.
Significance level, false positive rate, or the alpha level, is the probability of finding the alternative to be true when the null hypothesis is true: ( significance ) := α := P r ( find H 1 | H 0 ) {\displaystyle ({\text{significance}}):=\alpha :=Pr({\text{find }}H_{1}|H_{0})} For example, when the test is a one-sided threshold test, then α = P r D ∼ H 0 ( t [ D ] > t threshold ) {\displaystyle \alpha =Pr_{D\sim H_{0}}(t[D]>t_{\text{threshold}})} where D ∼ H 0 {\displaystyle D\sim H_{0}} means "the data is sampled from H 0 {\displaystyle H_{0}} ". Statistical power, true positive rate, is the probability of finding the alternative to be true when the alternative hypothesis is true: ( power ) := 1 − β := P r ( find H 1 | H 1 ) {\displaystyle ({\text{power}}):=1-\beta :=Pr({\text{find }}H_{1}|H_{1})} where β {\displaystyle \beta } is also called the false negative rate. For example, when the test is a one-sided threshold test, then 1 − β = P r D ∼ H 1 ( t [ D ] > t threshold ) {\displaystyle 1-\beta =Pr_{D\sim H_{1}}(t[D]>t_{\text{threshold}})} . Given a statistical test and a data set D {\displaystyle D} , the corresponding p-value is the probability that the test statistic is at least as extreme, conditional on H 0 {\displaystyle H_{0}} . For example, for a one-sided threshold test, p [ D ] = P r D ′ ∼ H 0 ( t [ D ′ ] > t [ D ] ) {\displaystyle p[D]=Pr_{D'\sim H_{0}}(t[D']>t[D])} If the null hypothesis is true, then the p-value is distributed uniformly on [ 0 , 1 ] {\displaystyle [0,1]} . Otherwise, it is typically peaked at p = 0.0 {\displaystyle p=0.0} and roughly exponential, though the precise shape of the p-value distribution depends on what the alternative hypothesis is. Because the p-values are distributed uniformly on [ 0 , 1 ] {\displaystyle [0,1]} under the null hypothesis, researchers can set any significance level α {\displaystyle \alpha } by computing the p-value, then output H 1 {\displaystyle H_{1}} if p [ D ] < α {\displaystyle p[D]<\alpha } . This is usually stated as "the null hypothesis is rejected at significance level α {\displaystyle \alpha } ", or " H 1 ( p < α ) {\displaystyle H_{1}\;(p<\alpha )} ", such as "smoking is correlated with cancer (p < 0.001)".
History The replication crisis dates to a number of events in the early 2010s. Felipe Romero identified four precursors to the crisis:
Social priming failures: In the early 2010s, two direct replication attempts failed to reproduce results from social psychologist John Bargh's much-cited "elderly-walking" study (originally published in 1996). This experiment was part of a series of three studies that had been widely cited throughout the years, was regularly taught in university courses, and had inspired many conceptual replications. These replication failures triggered intense disagreement between replication researchers and the original authors. Notably, many of the conceptual replications of the original studies also failed to replicate in subsequent direct replications. Experiments on extrasensory perception: Social psychologist Daryl Bem conducted a series of experiments supposedly providing evidence for the controversial phenomenon of extrasensory perception. Bem faced substantial criticism of his study's methodology. Reanalysis of his data found no evidence for extrasensory perception. The experiment also failed to replicate in subsequent direct replications. According to Romero, what the community found particularly upsetting was that many of the flawed procedures and statistical tools used in Bem's studies were part of common research practice in psychology. Biomedical replication failures: Scientists from biotech companies Amgen and Bayer Healthcare reported alarmingly low replication rates (11–20%) of landmark findings in preclinical oncological research. P-hacking studies and questionable research practices: Since the late 2000s, a number of studies in metascience showed how commonly adopted practices in many scientific fields, such as exploiting the flexibility of the process of data collection and reporting, could greatly increase the probability of false positive results. These studies suggested how a significant proportion of published literature in several scientific fields could be nonreplicable research. This series of events generated a great deal of skepticism about the validity of existing research in light of widespread methodological flaws and failures to replicate findings. This led prominent scholars to declare a "crisis of confidence" in psychology and other fields, and the ensuing situation came to be known as the "replication crisis". Although the beginning of the replication crisis can be traced to the early 2010s, some authors point out that concerns about replicability and research practices in the social sciences had been expressed much earlier. Romero notes that authors voiced concerns about the lack of direct replications in psychological research in the late 1960s and early 1970s. He also writes that certain studies in the 1990s were already reporting that journal editors and reviewers are generally biased against publishing replication studies. In the social sciences, the blog Data Colada (whose three authors coined the term "p-hacking" in a 2014 paper) has been credited with contributing to the start of the replication crisis. University of Virginia professor and cognitive psychologist Barbara Spellman has written that many criticisms of research practices and concerns about replicability of research are not new. She reports that between the late 1950s and the 1990s, scholars were already expressing concerns about a possible crisis of replication, a suspiciously high rate of positive findings, questionable research practices, the effects of publication bias, issues with statistical power, and bad standards of reporting. Spellman also identifies reasons that the reiteration of these criticisms and concerns in recent years led to a full-blown crisis and challenges to the status quo. First, technological improvements facilitated conducting and disseminating replication studies, and analyzing large swaths of literature for systemic problems. Second, the research community's increasing size and diversity made the work of established members more easily scrutinized by other community members unfamiliar with them. According to Spellman, these factors, coupled with increasingly limited resources and misaligned incentives for doing scientific work, led to a crisis in psychology and other fields. According to Andrew Gelman, the works of Paul Meehl, Jacob Cohen, and Tversky and Kahneman in the 1960s-70s were early warnings of replication crisis. In discussing the origins of the problem, Kahneman himself noted historical precedents in subliminal perception and dissonance reduction replication failures. It had been repeatedly pointed out since 1962 that most psychological studies have low power (true positive rate), but low power persisted for 50 years, indicating a structural and persistent problem in psychological research.
Prevalence
In psychology Several factors have combined to put psychology at the center of the conversation. Some areas of psychology once considered solid, such as social priming and ego depletion, have come under increased scrutiny due to failed replications. Much of the focus has been on social psychology, although other areas of psychology such as clinical psychology, developmental psychology, and educational research have also been implicated. In August 2015, the first open empirical study of reproducibility in psychology was published, called The Reproducibility Project: Psychology. Coordinated by psychologist Brian Nosek, researchers redid 100 studies in psychological science from three high-ranking psychology journals (Journal of Personality and Social Psychology, Journal of Experimental Psychology: Learning, Memory, and Cognition, and Psychological Science). Of 97 original studies with significant effects, 36% replicated successfully (p value below 0.05), with effect sizes averaging half the original magnitude. Among non-replications, 25% directly contradicted the original while 49% were inconclusive due to underpowered designs. The same paper examined the reproducibility rates and effect sizes by journal and discipline. Study replication rates were 23% for the Journal of Personality and Social Psychology, 48% for Journal of Experimental Psychology: Learning, Memory, and Cognition, and 38% for Psychological Science. Studies in the field of cognitive psychology had a higher replication rate (50%) than studies in the field of social psychology (25%). This inconclusiveness reflected inadequate statistical power: replication samples were approximately 40% the size of the originals. A study published in 2018 in Nature Human Behaviour replicated 21 social and behavioral science papers from Nature and Science, finding that only about 62% could successfully reproduce original results. Similarly, in a study conducted under the auspices of the Center for Open Science, a team of 186 researchers from 60 laboratories (representing 36 nationalities from six continents) conducted replications of 28 classic and contemporary findings in psychology. The study's focus was not only whether the original papers' findings replicated but also the extent to which findings varied as a function of variations in samples and contexts. Overall, 50% of findings failed to replicate despite large sample sizes. When findings did replicate, they consistently replicated across most samples. When they failed, they consistently failed across contexts, suggesting contextual sensitivity was not the primary driver of replication failures. This evidence is inconsistent with a proposed explanation that failures to replicate in psychology are likely due to changes in the sample between the original and replication study. Results of a 2022 study suggest that many earlier brain–phenotype studies ("brain-wide association studies" (BWAS)) produced invalid conclusions as the replication of such studies requires samples from thousands of individuals due to small effect sizes.
In medicine
Of 49 medical studies from 1990 to 2003 with more than 1000 citations, 92% found that the studied therapies were effective. Of these studies, 16% were contradicted by subsequent studies, 16% had found stronger effects than did subsequent studies, 44% were replicated, and 24% remained largely unchallenged. A 2011 analysis by researchers with pharmaceutical company Bayer found that, at most, a quarter of Bayer's in-house findings replicated the original results. But the analysis of Bayer's results found that the results that did replicate could often be successfully used for clinical applications. In a 2012 paper, C. Glenn Begley, a biotech consultant working at Amgen, and Lee Ellis, a medical researcher at the University of Texas, found that only 11% of 53 pre-clinical cancer studies had replications that could confirm conclusions from the original studies. In late 2021, The Reproducibility Project: Cancer Biology examined 53 top papers about cancer published between 2010 and 2012 and showed that among studies that provided sufficient information to be redone, the effect sizes were 85% smaller on average than the original findings. A survey of cancer researchers found that half of them had been unable to reproduce a published result. Another report estimated that almost half of randomized controlled trials contained flawed data (based on the analysis of anonymized individual participant data (IPD) from more than 150 trials).
In other disciplines
In nutrition science In nutrition science, for most food ingredients, there were studies that found that the ingredient has an effect on cancer risk. Specifically, out of a random sample of 50 ingredients from a cookbook, 80% had articles reporting on their cancer risk. Statistical significance decreased for meta-analyses.
In economics Economics has lagged behind other social sciences and psychology in its attempts to assess replication rates and increase the number of studies that attempt replication. A 2016 study in the journal Science replicated 18 experimental studies published in two leading economics journals, The American Economic Review and the Quarterly Journal of Economics, between 2011 and 2014. It found that approximately 61% of studies successfully replicated, though the replicated effect sizes were only 66% of the original reported effect sizes on average, suggesting that the original studies' effect sizes were inflated. About 20% of studies published in The American Economic Review are contradicted by other studies despite relying on the same or similar data sets. A study of empirical findings in the Strategic Management Journal found that about 30% of 27 retested articles showed statistically insignificant results for previously significant findings, whereas about 4% showed statistically significant results for previously insignificant findings.
In water resource management A 2019 study in Scientific Data estimated with 95% confidence that of 1,989 articles on water resources and management published in 2017, study results might be reproduced for only 0.6% to 6.8%, largely because the articles did not provide sufficient information to allow for replication.
In research on and with large language models The rapid adoption of large language models (LLMs) after 2022 introduced a distinct set of reproducibility concerns, affecting both research on LLMs and research that uses them as tools. One difficulty is that empirical findings about how LLMs behave are often hard to reproduce. Commercial models are updated or retired without notice, their training data and weights are usually undisclosed, and their outputs can shift from one run to the next even when settings are held constant, so a result reported once may not hold when the same prompts are tried later. Researchers have warned that this combination of closed, fast-changing systems and sensitivity to small changes in prompt wording could produce a wave of findings that fail to replicate. Reproducibility is also strained when LLMs are used as instruments within other fields. Reviews of machine learning research have found that many papers cannot be reproduced even in principle, because the code, data, exact hyperparameters, random seeds, or model versions go unreported, while reliance on proprietary models and datasets keeps others from verifying the results. A recurring hazard across these studies is data leakage, in which information from the test set inadvertently enters training and inflates apparent performance; it has been singled out as a leading cause of irreproducible claims in machine-learning-based science. A 2025 review of 640 papers that applied LLMs to software engineering tasks found persistent gaps in the availability of artefacts, the specification of computing environments, and documentation, and called the situation a reproducibility crisis within that subfield. A further worry is that generative AI can corrupt the data on which behavioral research depends. Online participant pools such as Prolific and Amazon Mechanical Turk are widely used in psychology, but some participants now lean on chatbots to answer open-ended questions, producing fluent text that hides inattention and can distort experimental results. Analyzing keystroke records from three studies run on Prolific in 2025, one team estimated notable rates of AI-assisted responding and cautioned that this kind of contamination erodes statistical power and undermines the validity of crowdsourced studies. In response, researchers have begun developing reporting standards tailored to LLM-based work. In 2026, a large consensus group published a reporting checklist in Nature Human Behaviour, known as GUIDE-LLM, intended to improve the transparency, reproducibility, and ethical accountability of LLM-based research in the behavioral sciences. The checklist asks authors to document which models and versions they used, how prompts and parameters were chosen, and what steps were taken to guard against the instability and contamination problems described above, so that other researchers stand a better chance of reproducing the work.
Across fields A 2016 survey by Nature on 1,576 researchers who took a brief online questionnaire on reproducibility found that more than 70% of researchers have tried and failed to reproduce another scientist's experiment results (including 87% of chemists, 77% of biologists, 69% of physicists and engineers, 67% of medical researchers, 64% of earth and environmental scientists, and 62% of all others), and more than half have failed to reproduce their own experiments. But fewer than 20% had been contacted by another researcher unable to reproduce their work. The survey found that fewer than 31% of researchers believe that failure to reproduce results means that the original result is probably wrong, although 52% agree that a significant replication crisis exists. Most researchers said they still trust the published literature. In 2010, Fanelli (2010) found that 91.5% of psychiatry/psychology studies confirmed the effects they were looking for, and concluded that the odds of this happening (a positive result) was around five times higher than in fields such as astronomy or geosciences. Fanelli argued that this is because researchers in "softer" sciences have fewer constraints to their conscious and unconscious biases. Early analysis of result-blind peer review, which is less affected by publication bias, has estimated that 61% of result-blind studies in biomedicine and psychology have led to null results, in contrast to an estimated 5% to 20% in earlier research. In 2021, a study conducted by University of California, San Diego found that papers that cannot be replicated are more likely to be cited. Nonreplicable publications are often cited more even after a replication study is published. In 2026, the SCORE Project study of 3,900 social-science papers between 2009 and 2018 found that only about half the results could be replicated.
Causes There are many proposed causes for the replication crisis.
Historical and sociological causes The replication crisis may be triggered by the "generation of new data and scientific publications at an unprecedented rate" that leads to "desperation to publish or perish" and failure to adhere to good scientific practice. Predictions of an impending crisis in the quality-control mechanism of science can be traced back several decades. Derek de Solla Price—considered the father of scientometrics, the quantitative study of science—predicted in his 1963 book Little Science, Big Science that science could reach "senility" as a result of its own exponential growth. Some present-day literature seems to vindicate this "overflow" prophecy, lamenting the decay in both attention and quality. Historian Philip Mirowski argues that the decline of scientific quality can be connected to its commodification, especially spurred by major corporations' profit-driven decision to outsource their research to universities and contract research organizations. Social systems theory, as expounded in the work of German sociologist Niklas Luhmann, inspires a similar diagnosis. This theory holds that each system, such as economy, science, religion, and media, communicates using its own code: true and false for science, profit and loss for the economy, news and no-news for the media, and so on. According to some sociologists, science's mediatization, commodification, and politicization, as a result of the structural coupling among systems, have led to a confusion of the original system codes.
Problems with the publication system in science
Publication bias Publication bias—the tendency to publish only positive, significant results—creates the "file drawer effect", where negative results remain unpublished. This produces misleading literature and biased meta-analyses. Only a very small proportion of academic journals in psychology and neurosciences explicitly welcomed submissions of replication studies in their aim and scope or instructions to authors. This does not encourage reporting on, or even attempts to perform, replication studies. Among 1,576 researchers Nature surveyed in 2016, only a minority had ever attempted to publish a replication, and several respondents who had published failed replications noted that editors and reviewers demanded that they play down comparisons with the original studies. An analysis of 4,270 empirical studies in 18 business journals from 1970 to 1991 reported that less than 10% of accounting, economics, and finance articles and 5% of management and marketing articles were replication studies. Publication bias is augmented by the pressure to publish and the author's own confirmation bias, and is an inherent hazard in the field, requiring a certain degree of skepticism on the part of readers. When publication bias is considered along with the fact that a majority of tested hypotheses might be false a priori, it is plausible that a considerable proportion of research findings might be false positives, as shown by metascientist John Ioannidis. In turn, a high proportion of false positives in the published literature can explain why many findings are nonreproducible. Another publication bias is that studies that do not reject the null hypothesis are scrutinized asymmetrically. For example, they are likely to be rejected as being difficult to interpret or having a Type II error. Studies that do reject the null hypothesis are not likely to be rejected for those reasons. In popular media, there is another element of publication bias: the desire to make research accessible to the public led to oversimplification and exaggeration of findings, creating unrealistic expectations and amplifying the impact of non-replications. In contrast, null results and failures to replicate tend to go unreported. This explanation may apply to power posing's replication crisis.
Mathematical errors Even high-impact journals have a significant fraction of mathematical errors in their use of statistics. For example, 11% of statistical results published in Nature and BMJ in 2001 are "incongruent", meaning that the reported p-value is mathematically different from what it should be if it were correctly calculated from the reported test statistic. These errors were likely from typesetting, rounding, and transcription errors. Among 157 neuroscience papers published in five top-ranking journals that attempt to show that two experimental effects are different, 78 erroneously tested instead for whether one effect is significant while the other is not, and 79 correctly tested for whether their difference is significantly different from 0.
"Publish or perish" culture
Academic "publish or perish" culture exacerbates publication bias. Intense pressure to publish in recognized journals, driven by hypercompetitive environments and bibliometric career evaluations, incentivizes researchers to prioritize publishable results over validity. According to Fanelli, this pushes scientists to employ a number of strategies aimed at making results "publishable". In the context of publication bias, this can mean adopting behaviors aimed at making results positive or statistically significant, often at the expense of their validity. According to Center for Open Science founder Brian Nosek and his colleagues, "publish or perish" culture created a situation whereby the goals and values of single scientists (e.g., publishability) are not aligned with the general goals of science (e.g., pursuing scientific truth). This is detrimental to the validity of published findings. Philosopher Brian D. Earp and psychologist Jim A. C. Everett argue that, although replication is in the best interests of academics and researchers as a group, features of academic psychological culture discourage replication by individual researchers. They argue that performing replications can be time-consuming, and take away resources from projects that reflect the researcher's original thinking. They are harder to publish, largely because they are unoriginal, and even when they can be published they are unlikely to be viewed as major contributions to the field. Replications "bring less recognition and reward, including grant money, to their authors". In his 1971 book Scientific Knowledge and Its Social Problems, philosopher and historian of science Jerome R. Ravetz predicted that science—in its progression from "little" science composed of isolated communities of researchers to "big" science or "techno-science"—would suffer major problems in its internal system of quality control. He recognized that the incentive structure for modern scientists could become dysfunctional, creating perverse incentives to publish any findings, however dubious. According to Ravetz, quality in science is maintained only when there is a community of scholars, linked by a set of shared norms and standards, who are willing and able to hold each other accountable.
Standards of reporting Certain publishing practices also make it difficult to conduct replications and to monitor the severity of the reproducibility crisis, for articles often come with insufficient descriptions for other scholars to reproduce the study. The Reproducibility Project: Cancer Biology showed that of 193 experiments from 53 top papers about cancer published between 2010 and 2012, only 50 experiments from 23 papers have authors who provided enough information for researchers to redo the studies, sometimes with modifications. None of the 193 papers examined had its experimental protocols fully described and replicating 70% of experiments required asking for key reagents. The aforementioned study of empirical findings in the Strategic Management Journal found that 70% of 88 articles could not be replicated due to a lack of sufficient information for data or procedures. In water resources and management, most of 1,987 articles published in 2017 were not replicable because of a lack of available information shared online. In studies of event-related potentials, only two-thirds the information needed to replicate a study were reported in a sample of 150 studies, highlighting that there are substantial gaps in reporting.
Procedural bias By the Duhem-Quine thesis, scientific results are interpreted by both a substantive theory and a theory of instruments. For example, astronomical observations depend both on the theory of astronomical objects and the theory of telescopes. A large amount of non-replicable research might accumulate if there is a bias of the following kind: faced with a null result, a scientist prefers to treat the data as saying the instrument is insufficient; faced with a non-null result, a scientist prefers to accept the instrument as good, and treat the data as saying something about the substantive theory.
Cultural evolution Smaldino and McElreath proposed a simple model for the cultural evolution of scientific practice. Each lab randomly decides to produce novel research or replication research, at different fixed levels of false positive rate, true positive rate, replication rate, and productivity (its "traits"). A lab might use more "effort", making the ROC curve more convex but decreasing productivity. A lab accumulates a score over its lifetime that increases with pub