Q 5 (10 points)
A commonly used model of the distribution of terms in a collection is
Zipf's law. It is stated as, if $t_1$ is the most common term in the collection,
$t_2$ is the next most common, and so on, then the collection frequency $cf_i$,
of the $i$th most common term is proportional to $1/i$. That is,
$cf_i \propto \frac{1}{i}$
So if the most frequent term occurs $cf_1$ times, then the second most
frequent term has half as many occurrences, the third most frequent
term a third as many occurrences, and so on. The intuition is that
frequency decreases very rapidly with rank.
Equivalently, we can write Zipf's law as $cf_i = ci^k$ or as $log\,cf_i = log\,c +$
k $log\,i$ where $k = -1$ and c is a constant (for English, $c = 0.1$).
If we ignored all occurrences of the five most frequent terms in the
collection, what proportion of term occurrences would be removed from
the collection?
