In natural language processing, GloVe (acronym for Global Vectors) is a model for distributed word representation. The model is an unsupervised learning algorithm for obtaining vector representations of words. This is achieved by mapping words into a meaningful space where the distance between words is related to semantic similarity. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space. As log-bilinear regression model for unsupervised learning of word representations, it combines the features of two model families, namely the global matrix factorization and local context window methods. It was developed as an open-source project at Stanford University and launched in 2014. It was designed as a competitor to word2vec, and the original paper noted multiple improvements of GloVe over word2vec. As of 2022, both approaches are outdated, and transformer-based models, such as BERT, which add multiple neural-network attention layers on top of a word embedding model similar to Word2vec, have come to be regarded as the state of the art in natural language processing (NLP).
Definition You shall know a word by the company it keeps (Firth, J. R. 1957:11)The idea of GloVe is to construct, for each word i {\displaystyle i} , two vectors w i , w ~ i {\displaystyle w_{i},{\tilde {w}}_{i}} , such that the relative positions of the vectors capture part of the statistical regularities of the word i {\displaystyle i} . The statistical regularity is defined as the co-occurrence probabilities. Words that resemble each other in meaning should also resemble each other in co-occurrence probabilities.
Word counting Let the vocabulary be V {\displaystyle V} , the set of all possible words (aka "tokens"). Punctuation is either ignored, or treated as vocabulary, and similarly for capitalization and other typographical details.
If two words occur close to each other, then we say that they occur in the context of each other. For example, if the context length is 3, then we say that in the following sentenceGloVe1, coined2 from3 Global4 Vectors5, is6 a7 model8 for9 distributed10 word11 representation12the word "model8" is in the context of "word11" but not the context of "representation12". A word is not in the context of itself, so "model8" is not in the context of the word "model8", although, if a word appears again in the same context, then it does count. Let X i j {\displaystyle X_{ij}} be the number of times that the word j {\displaystyle j} appears in the context of the word i {\displaystyle i} over the entire corpus. For example, if the corpus is just "I don't think that that is a problem." we have X that , that = 2 {\displaystyle X_{{\text{that}},{\text{that}}}=2} since the first "that" appears in the second one's context, and vice versa. Let X i = ∑ j ∈ V X i j {\displaystyle X_{i}=\sum _{j\in V}X_{ij}} be the number of words in the context of all instances of word i {\displaystyle i} . By counting, we have X i = 2 × ( context size ) × # ( occurrences of word i ) {\displaystyle X_{i}=2\times ({\text{context size}})\times \#({\text{occurrences of word }}i)} (except for words occurring right at the start and end of the corpus)
… excerpt ends here. Continue reading the full article.
