Neural language models have seen a dramatic increase in size in the last years. While many still advocate that ‘bigger is better’, work in model distillation has shown that the number of parameters used by very large networks is actually more than what is required for state-of-the-art performance. This prompts an obvious question: can we build smaller models from scratch, rather than going through the inefficient process of training at scale and subsequently reducing model size. In this paper, we investigate the behaviour of a biologically inspired algorithm, based on the fruit fly’s olfactory system. This algorithm has shown good performance in the past on the task of learning word embeddings. We now put it to the test on the task of semantic hashing. Specifically, we compare the fruit fly to a standard binary network on the task of generating locality-sensitive hashes for text documents, measuring both task performance and energy consumption. Our results indicate that the two algorithms have complementary strengths while showing similar electricity usage.

Algorithmic Diversity and Tiny Models: Comparing Binary Networks and the Fruit Fly Algorithm on Document Representation Tasks / Ceron, Tanise; Truong, Nhut; Herbelot, Aurelie. - ELETTRONICO. - (2022), pp. 17-28. (Intervento presentato al convegno SustaiNLP tenutosi a Abu Dhabi, United Arab Emirates (Hybrid) nel 7th December 2022) [10.18653/v1/2022.sustainlp-1.4].

Algorithmic Diversity and Tiny Models: Comparing Binary Networks and the Fruit Fly Algorithm on Document Representation Tasks

Truong, Nhut
Co-primo
;
Herbelot, Aurelie
Ultimo
2022-01-01

Abstract

Neural language models have seen a dramatic increase in size in the last years. While many still advocate that ‘bigger is better’, work in model distillation has shown that the number of parameters used by very large networks is actually more than what is required for state-of-the-art performance. This prompts an obvious question: can we build smaller models from scratch, rather than going through the inefficient process of training at scale and subsequently reducing model size. In this paper, we investigate the behaviour of a biologically inspired algorithm, based on the fruit fly’s olfactory system. This algorithm has shown good performance in the past on the task of learning word embeddings. We now put it to the test on the task of semantic hashing. Specifically, we compare the fruit fly to a standard binary network on the task of generating locality-sensitive hashes for text documents, measuring both task performance and energy consumption. Our results indicate that the two algorithms have complementary strengths while showing similar electricity usage.
2022
Proceedings of The Third Workshop on Simple and Efficient Natural Language Processing
Cambridge, Massachusetts
Association for Computational Linguistics
Ceron, Tanise; Truong, Nhut; Herbelot, Aurelie
Algorithmic Diversity and Tiny Models: Comparing Binary Networks and the Fruit Fly Algorithm on Document Representation Tasks / Ceron, Tanise; Truong, Nhut; Herbelot, Aurelie. - ELETTRONICO. - (2022), pp. 17-28. (Intervento presentato al convegno SustaiNLP tenutosi a Abu Dhabi, United Arab Emirates (Hybrid) nel 7th December 2022) [10.18653/v1/2022.sustainlp-1.4].
File in questo prodotto:
File Dimensione Formato  
2022.sustainlp-1.4.pdf

accesso aperto

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Creative commons
Dimensione 521.76 kB
Formato Adobe PDF
521.76 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/387909
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
  • OpenAlex ND
social impact