The widespread way to represent words in a text to perform natural language processing is through word embeddings. The challenge is to represent more accurately as possible the meaning of a word, which may have many nuances of meanings. Nowadays, many models create a word embedding considering the words appearing in the text near the token to embed. The procedure requires large quantities and varieties of corpora to cover the natural language words. Furthermore, the word embedding reconstruction of a token, unobserved during the training phase, is still laborious or inaccurate. We propose the Def2Vec model, which generates word representations using a reduced and complete dataset, named a dictionary, and new word embeddings efficiently. Its completeness also allowed us to learn word embeddings of specific lexicons. Furthermore, the dictionary definitions are more objective than generic datasets, which may be biased or contain stereotypes. The Def2Vec model exploits the bag-of-words representation to encode the dictionary definitions and the TF-IDF to normalize the bag-of-words vectors. Finally, it exploits the SVD factorization to extract the final word embeddings. The SVD factorization is also the key in the procedure to reconstruct the new word embeddings. It allowed us to build word embeddings of unobserved words without the need to reconstruct, again, all the word embeddings already computed. We tested the Def2Vec model in the word embeddings construction for the Part-Of-Speech tagging, Named Entity Recognition, Chunking, and Semantic Text Similarity tasks. We performed the same evaluations when testing the validity of the new word embeddings generation. The Def2Vec word embeddings obtained performances comparable to, and sometimes even better than, the state-of-the-art models, like Word2Vec, FastText, and GloVe. The reconstructed Def2vec word embeddings obtained scores equal to and even better than the word embeddings generated considering the whole starting dataset. The Def2Vec is a model for creating word embeddings, exploitable for text analysis applications, and for reconstructing briefly accurate new word embeddings.
Il sistema più diffuso per rappresentare le parole di un testo al fine di elaborare dei contenuti in linguaggio naturale è l'uso dei word embeddings. La sfida è di creare delle rappresentazioni il più accurate possibili dell’accezione di una parola, la quale può avere molte sfumature di significati. Attualmente molti modelli creano un word embedding considerando i termini che appaiono nel testo vicini alla parola da rappresentare. La procedura richiede grandi quantità e varietà di dati per coprire le parole di un linguaggio naturale. Inoltre, la ricostruzione del word embedding di un termine, che non è stato osservato durante la fase di apprendimento, è ancora laboriosa e inaccurata. Noi proponiamo il modello Def2Vec che genera word embeddings utilizzando un dizionario come insieme di dati, che è ridotto, completo e contiene definizioni oggettive. Queste proprietà permettono di ottenere una rappresentazione delle parole rare e prevenire l’assorbimento di bias e stereotipi nelle rappresentazioni delle parole. Il modello Def2Vec rappresenta le definizioni del dizionario come bag-of-words e le normalizza con la funzione di peso TF-IDF. Infine, estrae i word embeddings finali sfruttando la decomposizione ai valori singolari. Quest’ultima è anche la chiave per la ricostruzione delle parole non osservate in precedenza, che permette di non dover ricostruire, nuovamente, tutti i word embeddings già prodotti. Abbiamo testato il modello Def2Vec nella costruzione di word embeddings per i compiti di Part-Of-Speech tagging, Named entity Recognition, Chunking e Semantic Textual Similarity. Abbiamo eseguito le stesse valutazioni per verificare l’attendibilità dei word embeddings ricostruiti. I word embeddings di Def2Vec hanno ottenuto prestazioni paragonabili e a volte migliori dei modelli dello stato dell'arte, come Word2Vec, FastText e GloVe. I word embeddings ricostruiti da Def2Vec hanno ottenuto punteggi uguali o superiori ai word embeddings generati considerando l'intero dataset di partenza. Def2Vec è un modello per la creazione di word embeddings, utilizzabile per le applicazioni che analizzano testi e per la costruzione in breve di nuovi word embeddings accurati.
Def2Vec : a model to extract word embeddings from dictionary definitions
Morazzoni, Irene
2020/2021
Abstract
The widespread way to represent words in a text to perform natural language processing is through word embeddings. The challenge is to represent more accurately as possible the meaning of a word, which may have many nuances of meanings. Nowadays, many models create a word embedding considering the words appearing in the text near the token to embed. The procedure requires large quantities and varieties of corpora to cover the natural language words. Furthermore, the word embedding reconstruction of a token, unobserved during the training phase, is still laborious or inaccurate. We propose the Def2Vec model, which generates word representations using a reduced and complete dataset, named a dictionary, and new word embeddings efficiently. Its completeness also allowed us to learn word embeddings of specific lexicons. Furthermore, the dictionary definitions are more objective than generic datasets, which may be biased or contain stereotypes. The Def2Vec model exploits the bag-of-words representation to encode the dictionary definitions and the TF-IDF to normalize the bag-of-words vectors. Finally, it exploits the SVD factorization to extract the final word embeddings. The SVD factorization is also the key in the procedure to reconstruct the new word embeddings. It allowed us to build word embeddings of unobserved words without the need to reconstruct, again, all the word embeddings already computed. We tested the Def2Vec model in the word embeddings construction for the Part-Of-Speech tagging, Named Entity Recognition, Chunking, and Semantic Text Similarity tasks. We performed the same evaluations when testing the validity of the new word embeddings generation. The Def2Vec word embeddings obtained performances comparable to, and sometimes even better than, the state-of-the-art models, like Word2Vec, FastText, and GloVe. The reconstructed Def2vec word embeddings obtained scores equal to and even better than the word embeddings generated considering the whole starting dataset. The Def2Vec is a model for creating word embeddings, exploitable for text analysis applications, and for reconstructing briefly accurate new word embeddings.| File | Dimensione | Formato | |
|---|---|---|---|
|
2021_10_morazzoni.pdf
non accessibile
Descrizione: Def2Vec: a Model to Extract Word Embeddings from Dictionary Definitions
Dimensione
3.16 MB
Formato
Adobe PDF
|
3.16 MB | Adobe PDF | Visualizza/Apri |
I documenti in POLITesi sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/10589/179715