New frontiers in the p-hacking practices: assessing the effect of data peeking practice on type-I error rate through computational modeling

Data-peeking, the practice of stopping data collection once significant results are obtained, poses a significant threat to the credibility of research by increasing the likelihood of observing and publishing false positive results. The effect of this practice on scientific reliability has traditionally been assessed through Monte-Carlo simulations, which suffer from some methodological issues (e.g., Monte–Carlo standard error is often overestimated). This paper presents a computational model that, through a series of convolutions between normal and truncated normal distributions, estimates the increase in false-positive rates due to data-peeking practice, overcoming the issues of the simulation method. The performance of the computational model was compared against Monte-Carlo simulations under different conditions, including alpha levels (0.05, 0.025, 0.01, and 0.001), sample sizes (40, 80, 160, and 320), and numbers of interim analyses (1 to 10). The results confirmed the computational model's accuracy and reliability, showing that computational values fell within two standard deviations of the simulated means. In conclusion, the proposed model not only presents an faster and error-reducing approach to estimating the effects of data-peeking practice but also provides guidelines for the development of similar models to examine other forms of p-hacking, thereby improving the understanding and study of their effects.