In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a number of appropriate quantitative metrics to assess the processes main properties. We then apply the proposed metrics and analysis framework to data sets from conference evaluation processes and we discuss the results implications and their eventual use toward improving the analyzed peer review processes. A number of unexpected results were found, in particular: (1) the low correlation between peer review outcome and impact in time of the accepted contributions and (2) the presence of a high level of randomness in the analyzed peer review processes

A Quantitative Analysis of Peer Review

Mirylenka, Katsiaryna;Casati, Fabio;Marchese, Maurizio
2011-01-01

Abstract

In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a number of appropriate quantitative metrics to assess the processes main properties. We then apply the proposed metrics and analysis framework to data sets from conference evaluation processes and we discuss the results implications and their eventual use toward improving the analyzed peer review processes. A number of unexpected results were found, in particular: (1) the low correlation between peer review outcome and impact in time of the accepted contributions and (2) the presence of a high level of randomness in the analyzed peer review processes
2011
Proceedings of ISSI 2011
AA. VV.
Leuven, Belgium
ISSI
9789081752701
A., Ragone; Mirylenka, Katsiaryna; Casati, Fabio; Marchese, Maurizio
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/100744
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? 2
  • OpenAlex ND
social impact