A word n-gram language model is a statistical model of language which calculates the probability of the next word in a sequence from a fixed size window of previous words. If one previous word is considered, it is a bigram model; if two words, a trigram model; if n − 1 words, an n-gram model. Special tokens are introduced to denote the start and end of a sentence ⟨ s ⟩ {\displaystyle \langle s\rangle } and ⟨ / s ⟩ {\displaystyle \langle /s\rangle } . To prevent a zero probability being assigned to unseen words, the probability of each seen word is slightly lowered to make room for the unseen words in a given corpus. To achieve this, various smoothing methods are used, from simple "add-one" smoothing (assigning a count of 1 to unseen n-grams, as an uninformative prior) to more sophisticated techniques, such as Good–Turing discounting or back-off models. Word n-gram models have largely been superseded by recurrent neural network–based models, which in turn have been superseded by Transformer-based models often referred to as large language models.
Unigram model
A special case, where n = 1, is called a unigram model. Probability of each word in a sequence is independent from probabilities of other word in the sequence. Each word's probability in the sequence is equal to the word's probability in an entire document.
P uni ( t 1 t 2 t 3 ) = P ( t 1 ) P ( t 2 ) P ( t 3 ) . {\displaystyle P_{\text{uni}}(t_{1}t_{2}t_{3})=P(t_{1})P(t_{2})P(t_{3}).}
The model consists of units, each treated as one-state finite automata. Words with their probabilities in a document can be illustrated as follows.
Total mass of word probabilities distributed across the document's vocabulary, is 1.
∑ word in doc P ( word ) = 1 {\displaystyle \sum _{\text{word in doc}}P({\text{word}})=1}
The probability generated for a specific query is calculated as
P ( query ) = ∏ word in query P ( word ) {\displaystyle P({\text{query}})=\prod _{\text{word in query}}P({\text{word}})}
Unigram models of different documents have different probabilities of words in it. The probability distributions from different documents are used to generate hit probabilities for each query. Documents can be ranked for a query according to the probabilities. Example of unigram models of two documents:
Bigram model In a bigram word (n = 2) language model, the probability of the sentence I saw the red house is approximated as
P ( I, saw, the, red, house ) ≈ P ( I ∣ ⟨ s ⟩ ) P ( saw ∣ I ) P ( the ∣ saw ) P ( red ∣ the ) P ( house ∣ red ) P ( ⟨ / s ⟩ ∣ house ) {\displaystyle P({\text{I, saw, the, red, house}})\approx P({\text{I}}\mid \langle s\rangle )P({\text{saw}}\mid {\text{I}})P({\text{the}}\mid {\text{saw}})P({\text{red}}\mid {\text{the}})P({\text{house}}\mid {\text{red}})P(\langle /s\rangle \mid {\text{house}})}
Trigram model In a trigram (n = 3) language model, the approximation is
P ( I, saw, the, red, house ) ≈ P ( I ∣ ⟨ s ⟩ , ⟨ s ⟩ ) P ( saw ∣ ⟨ s ⟩ , I ) P ( the ∣ I, saw ) P ( red ∣ saw, the ) P ( house ∣ the, red ) P ( ⟨ / s ⟩ ∣ red, house ) {\displaystyle P({\text{I, saw, the, red, house}})\approx P({\text{I}}\mid \langle s\rangle ,\langle s\rangle )P({\text{saw}}\mid \langle s\rangle ,I)P({\text{the}}\mid {\text{I, saw}})P({\text{red}}\mid {\text{saw, the}})P({\text{house}}\mid {\text{the, red}})P(\langle /s\rangle \mid {\text{red, house}})}
Note that the context of the first n – 1 n-grams is filled with start-of-sentence markers, typically denoted <s>. Additionally, without an end-of-sentence marker, the probability of an ungrammatical sequence *I saw the would always be higher than that of the longer sentence I saw the red house.
… excerpt ends here. Continue reading the full article.
