Lexical similarity data, quantifying the “proximity” of languages based on the similarity of their lexicons, has been increasingly used to estimate the cross-lingual reusability of language resources, for tasks such as bilingual lexicon induction or cross-lingual transfer. Existing similarity data, however, originates from the field of comparative linguistics, computed from very small expert-curated vocabularies that are not supposed to be representative of modern lexicons. We explore a different, fully automated approach to lexical similarity computation, based on an existing 8-million-entry cognate database created from online lexicons orders of magnitude larger than the word lists typically used in linguistics. We compare our results to earlier efforts, and automatically produce intuitive visualizations that have traditionally been hand-crafted. With a new, freely available database of over 27 thousand language pairs over 331 languages, we hope to provide more relevant data to cross-lingual NLP applications, as well as material for the synchronic study of contemporary lexicons.

A Database and Visualization of the Similarity of Contemporary Lexicons / Bella, Gábor; Batsuren, Khuyagbaatar; Giunchiglia, Fausto. - 12848:(2021), pp. 95-104. ((Intervento presentato al convegno TSD 2021 tenutosi a Olomouc, Czech Republic nel 6th-9th September 2021 [10.1007/978-3-030-83527-9_8].

A Database and Visualization of the Similarity of Contemporary Lexicons

Bella, Gábor;Batsuren, Khuyagbaatar;Giunchiglia, Fausto
2021

Abstract

Lexical similarity data, quantifying the “proximity” of languages based on the similarity of their lexicons, has been increasingly used to estimate the cross-lingual reusability of language resources, for tasks such as bilingual lexicon induction or cross-lingual transfer. Existing similarity data, however, originates from the field of comparative linguistics, computed from very small expert-curated vocabularies that are not supposed to be representative of modern lexicons. We explore a different, fully automated approach to lexical similarity computation, based on an existing 8-million-entry cognate database created from online lexicons orders of magnitude larger than the word lists typically used in linguistics. We compare our results to earlier efforts, and automatically produce intuitive visualizations that have traditionally been hand-crafted. With a new, freely available database of over 27 thousand language pairs over 331 languages, we hope to provide more relevant data to cross-lingual NLP applications, as well as material for the synchronic study of contemporary lexicons.
Text, Speech, and Dialogue: 24th International Conference Proceedings
Cham, CH
Springer Verlag
978-3-030-83526-2
978-3-030-83527-9
Bella, Gábor; Batsuren, Khuyagbaatar; Giunchiglia, Fausto
A Database and Visualization of the Similarity of Contemporary Lexicons / Bella, Gábor; Batsuren, Khuyagbaatar; Giunchiglia, Fausto. - 12848:(2021), pp. 95-104. ((Intervento presentato al convegno TSD 2021 tenutosi a Olomouc, Czech Republic nel 6th-9th September 2021 [10.1007/978-3-030-83527-9_8].
File in questo prodotto:
File Dimensione Formato  
Lexical_Similarity_TSD(1).pdf

embargo fino al 31/12/2022

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 2.01 MB
Formato Adobe PDF
2.01 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: http://hdl.handle.net/11572/313144
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? ND
social impact