Bad Research Practices: Consequences, and Detection of Questionable Interim Analysis

The general topic of this Ph.D. thesis regards "Questionable Research Practices" (QRPs) and their effect on the replicability crisis in psychology and social sciences. QRPs are a set of bad research practices carried out to increase the likelihood of obtaining a significant result by providing some sort of manipulation of collected data. QRPs are related to the so-called replicability crisis, defined as the inability to replicate the results of an increasing number of studies. The proposed dissertation focuses on a specific form of QRPs named "Questionable Interim Analysis" (QIA). QIA implies that the data collection stopping rule depends on the previous interim analysis result. While the effect of this practice on the growth of false positive results is well known, its impact on the replicability of studies is still unclear. Furthermore, an objective tool to assess the incidence of this practice in a single study is, to our knowledge, still missing. In this project, we tried to overcome these limitations by evaluating the effects of QRPs on replication success and by proposing a formal tool to resize and control the QIA effects on statistical results. This document begins with a brief overview on the replicability crisis, QRPs, and the methods for identifying these practices in the literature (Chapter 1). Chapter 2 introduces a methodology to clarify the effect of some QRPs (cherry picking, questionable interim analyses, questionable inclusion of covariates, and questionable subgroup analyses) on replicability and determinates which replication success metric performs better in presence of these practices. The results show that the metric based on the golden sceptical p-value maintains low values of false positive replication success when QRPs are assumed. Chapter 3 presents a new probabilistic model to identify the incidence of QIA practice in the literature. It provides information about the larger or lower probability of this practice by estimating the ratio between the probability that a published result was obtained on the basis of a QIA practice or, alternatively, by a standard data collection approach. In Chapter 4, we show an application of the probabilistic models in the context of simulation studies. In particular, we illustrate how the new proposal improves some widely used simulation approaches to assess the false positive results in the context of QIA practices. Finally, in Chapter 5 the proposed models are discussed in terms of their utilities, limitations, and possible future extensions.