1. Suppose we have a small corpus of text with the following sentences:
"I love machine learning."
"She loves machine learning."
"They learn machine language."
"I learn to code."
From this corpus, we want to compute the bigram probabilities. However, given the small size of the
dataset, we decide to use Add-k smoothing to handle potential zero probabilities for unseen bigrams.
Task:
1. Compute Raw Bigram Probabilities:
Without smoothing, calculate the probability of the bigram "love machine".
The formula for computing the raw bigram probabilities is:
$P(w_i|w_{i-1}) = \frac{C(w_{i-1}, w_i)}{C(w_{i-1})}$
Where:
$\bullet P(w_i|w_{i-1})$ is the probability of word $w_i$ following word $w_{i-1}$.
$\bullet C(w_{i-1}, w_i)$ is the count of the bigram $w_{i-1} w_i$ in the corpus.
$\bullet C(w_{i-1})$ is the count of the preceding word $w_{i-1}$ in the corpus.
2. Apply Add-k Smoothing:
Use Add-k smoothing to calculate the probability of the same bigram "love machine". For this
exercise, use k=1.
Remember the formula for Add-k smoothing for bigrams:
$P(w_i|w_{i-1}) = \frac{C(w_{i-1}, w_i) + k}{C(w_{i-1}) + kV}$
Where:
$\bullet C(w_{i-1}, w_i)$ is the count of the bigram $w_{i-1} w_i$.
$\bullet C(w_{i-1})$ is the count of the preceding word $w_{i-1}$.
$\bullet V$ is the vocabulary size.
Problem:
1. Compare the raw bigram probability from step 1 with the smoothed probability from step 2.
2. How would the probability change if k was set to a value greater than 1?
3. Compute the smoothed bigram probability for an unseen bigram, for instance, "love code". How
does Add-k smoothing help in this scenario?
