The statistical analysis of data stemming from dynamical systems, including, but not limited to, time series, routinely relies on the estimation of information theoretical quantities, most notably Shannon entropy. To this purpose, possibly the most widespread tool is provided by the so-called plug-in estimator, whose statistical properties in terms of bias and variance were investigated since the first decade after the publication of Shannon's seminal works. In the case of an underlying multinomial distribution, while the bias can be evaluated by knowing support and data set size, variance is far more elusive. The aim of the present work is to investigate, in the multinomial case, the statistical properties of an estimator of a parameter that describes the variance of the plug-in estimator of Shannon entropy. We then exactly determine the probability distributions that maximize that parameter. The results presented here allow one to set upper limits to the uncertainty of entropy assessments under the hypothesis of memoryless underlying stochastic processes.

Estimating the variance of Shannon entropy / Ricci, Leonardo; Perinelli, Alessio; Castelluzzo, Michele. - In: PHYSICAL REVIEW. E. - ISSN 2470-0045. - ELETTRONICO. - 104:2(2021), pp. 024220.1-024220.8. [10.1103/PhysRevE.104.024220]

Estimating the variance of Shannon entropy

Ricci, Leonardo;Perinelli, Alessio;Castelluzzo, Michele
2021-01-01

Abstract

The statistical analysis of data stemming from dynamical systems, including, but not limited to, time series, routinely relies on the estimation of information theoretical quantities, most notably Shannon entropy. To this purpose, possibly the most widespread tool is provided by the so-called plug-in estimator, whose statistical properties in terms of bias and variance were investigated since the first decade after the publication of Shannon's seminal works. In the case of an underlying multinomial distribution, while the bias can be evaluated by knowing support and data set size, variance is far more elusive. The aim of the present work is to investigate, in the multinomial case, the statistical properties of an estimator of a parameter that describes the variance of the plug-in estimator of Shannon entropy. We then exactly determine the probability distributions that maximize that parameter. The results presented here allow one to set upper limits to the uncertainty of entropy assessments under the hypothesis of memoryless underlying stochastic processes.
2021
2
Ricci, Leonardo; Perinelli, Alessio; Castelluzzo, Michele
Estimating the variance of Shannon entropy / Ricci, Leonardo; Perinelli, Alessio; Castelluzzo, Michele. - In: PHYSICAL REVIEW. E. - ISSN 2470-0045. - ELETTRONICO. - 104:2(2021), pp. 024220.1-024220.8. [10.1103/PhysRevE.104.024220]
File in questo prodotto:
File Dimensione Formato  
PhysRevE_2021_104_024220_Ricci_Perinelli_Castelluzzo.pdf

Solo gestori archivio

Descrizione: Articolo scientifico
Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 2.41 MB
Formato Adobe PDF
2.41 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/315025
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 8
  • ???jsp.display-item.citation.isi??? 5
social impact