The present work constitutes an attempt to investigate the relational structures learnt by mBERT, a multilingual transformer-based network, with respect to different cross-linguistic regularities proposed in the fields of theoretical and quantitative linguistics. We pursued this objective by relying on a zero-shot transfer experiment, evaluating the model’s ability to generalize its native task to artificial languages that could either respect or violate some proposed language universal, and comparing its performance to the output of BERT, a monolingual model with an identical configuration. We created four artificial corpora through a Probabilistic Context-Free Grammar by manipulating the distribution of tokens and the structure of their dependency relations. We showed that while both models were favoured by a Zipfian distribution of the tokens and by the presence of head-dependency type structures, the multilingual transformer network exhibited a stronger reliance on hierarchical cues compared to its monolingual counterpart.

Multilingualism Encourages Recursion: a Transfer Study with mBERT / De Varda, Andrea Gregor; Zamparelli, Roberto. - ELETTRONICO. - (2022), pp. 1-10. (Intervento presentato al convegno SIGTYP tenutosi a online nel Luglio 2022) [10.18653/v1/2022.sigtyp-1.1].

Multilingualism Encourages Recursion: a Transfer Study with mBERT

Zamparelli, Roberto
2022-01-01

Abstract

The present work constitutes an attempt to investigate the relational structures learnt by mBERT, a multilingual transformer-based network, with respect to different cross-linguistic regularities proposed in the fields of theoretical and quantitative linguistics. We pursued this objective by relying on a zero-shot transfer experiment, evaluating the model’s ability to generalize its native task to artificial languages that could either respect or violate some proposed language universal, and comparing its performance to the output of BERT, a monolingual model with an identical configuration. We created four artificial corpora through a Probabilistic Context-Free Grammar by manipulating the distribution of tokens and the structure of their dependency relations. We showed that while both models were favoured by a Zipfian distribution of the tokens and by the presence of head-dependency type structures, the multilingual transformer network exhibited a stronger reliance on hierarchical cues compared to its monolingual counterpart.
2022
Proceedings of the 4th Workshop on Research in Computational Linguistic Typology and Multilingual NLP
Seattle, Washington
Association for Computational Linguistics
De Varda, Andrea Gregor; Zamparelli, Roberto
Multilingualism Encourages Recursion: a Transfer Study with mBERT / De Varda, Andrea Gregor; Zamparelli, Roberto. - ELETTRONICO. - (2022), pp. 1-10. (Intervento presentato al convegno SIGTYP tenutosi a online nel Luglio 2022) [10.18653/v1/2022.sigtyp-1.1].
File in questo prodotto:
File Dimensione Formato  
DeVarda-Zamparelli-Recursion_in_mBERT-SIGTYP-ACL2022.pdf

accesso aperto

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Creative commons
Dimensione 149.49 kB
Formato Adobe PDF
149.49 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/353285
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact