Probability and Computing

Michael Mitzenmacher , Eli Upfal

Chapter 4 Chernoff Bounds - all with Video Answers

Educators

Chapter Questions

01:30

Problem 1

Alice and Bob play checkers often. Alice is a better player, so the probability that she wins any given game is $0.6$, independent of all other games. They decide to play a tournament of $n$ games. Bound the probability that Alice loses the tournament using a Chernoff bound.

Nick Johnson

Numerade Educator

05:54

Problem 2

We have a standard six-sided die. Let $X$ be the number of times that a 6 occurs over $n$ throws of the die. Let $p$ be the probability of the event $X \geq n / 4$. Compare the best upper bounds on $p$ that you can obtain using Markov's inequality, Chebyshev's inequality, and Chernoff bounds.

Ahmad Reda

Numerade Educator

03:11

Problem 3

(a) Determine the moment generating function for the binomial random variable $B(n, p)$.
(b) Let $X$ be a $B(n, p)$ random variable and $Y$ a $B(m, p)$ random variable, where $X$ and $Y$ are independent. Use part (a) to determine the moment generating function of $X+Y$.
(c) What can we conclude from the form of the moment generating function of $X+Y ?$

Aimal Hassan

Numerade Educator

02:49

Problem 4

Determine the probability of obtaining 55 or more heads when flipping a fair coin 100 times by an explicit calculation, and compare this with the Chernoff bound. Do the same for 550 or more heads in 1000 flips.

Jerrah Biggerstaff

Numerade Educator

01:49

Problem 5

We plan to conduct an opinion poll to find out the percentage of people in a community who want its president impeached. Assume that every person answers either yes or no. If the actual fraction of people who want the president impeached is $p$, we want to find an estimate $X$ of $p$ such that
$$
\operatorname{Pr}(|X-p| \leq \varepsilon p)>1-\delta
$$
for a given $\varepsilon$ and $\delta$, with $0<\varepsilon, \delta<1$.
We query $N$ people chosen independently and uniformly at random from the community and output the fraction of them who want the president impeached. How large should $N$ be for our result to be a suitable estimator of $p$ ? Use Chernoff bounds, and express $N$ in terms of $p, \varepsilon$, and $\delta$. Calculate the value of $N$ from your bound if $\varepsilon=0.1$ and $\delta=0.05$ and if you know that $p$ is between $0.2$ and $0.8$.

Sheryl Ezze

Numerade Educator

06:13

Problem 6

(a) In an election with two candidates using paper ballots, each vote is independently misrecorded with probability $p=0.02$. Use a Chernoff bound to bound the probability that more than $4 \%$ of the votes are misrecorded in an election of $1,000,000$ ballots.
(b) Assume that a misrecorded ballot always counts as a vote for the other candidate. Suppose that candidate A received 510,000 votes and that candidate B received 490,000 votes. Use Chernoff bounds to bound the probability that candidate B wins the election owing to misrecorded ballots. Specifically, let $X$ be the number of votes for candidate $\mathrm{A}$ that are misrecorded and let $Y$ be the number of votes for candidate $\mathrm{B}$ that are misrecorded. Bound $\operatorname{Pr}((X>k) \cap(Y<\ell))$ for suitable choices of $k$ and $\ell$.

Abigail Martyr

Numerade Educator

02:10

Problem 7

Throughout the chapter we implicitly assumed the following extension of the Chernoff bound. Prove that it is true.

Let $X=\sum_{i=1}^{n} X_{i}$, where the $X_{i}$ are independent $0-1$ random variables. Let $\mu=$ $\mathbf{E}[X]$. Choose any $\mu_{L}$ and $\mu_{H}$ such that $\mu_{L} \leq \mu \leq \mu_{H}$. Then, for any $\delta>0$,
$$
\operatorname{Pr}\left(X \geq(1+\delta) \mu_{H}\right) \leq\left(\frac{\mathrm{e}^{\delta}}{(1+\delta)^{(1+\delta)}}\right)^{\mu_{H}}
$$
Similarly, for any $0<\delta<1$,
$$
\operatorname{Pr}\left(X \leq(1-\delta) \mu_{L}\right) \leq\left(\frac{\mathrm{e}^{-\delta}}{(1-\delta)^{(1-\delta)}}\right)^{\mu_{L}}
$$

Ameer Said

Numerade Educator

View

Problem 8

We show how to construct a random permutation $\pi$ on $[1, n]$, given a black box that outputs numbers independently and uniformly at random from $[1, k]$ where $k \geq n$. If we compute a function $f:[1, n] \rightarrow[1, k]$ with $f(i) \neq f(j)$ for $i \neq j$, this yields a permutation; simply output the numbers $[1, n]$ according to the order of the $f(i)$ values. To construct such a function $f$, do the following for $j=1, \ldots, n:$ choose $f(j)$ by repeatedly obtaining numbers from the black box and setting $f(j)$ to the first number found such that $f(j) \neq f(i)$ for $i<j$.

Prove that this approach gives a permutation chosen uniformly at random from all permutations. Find the expected number of calls to the black box that are needed when $k=n$ and $k=2 n$. For the case $k=2 n$, argue that the probability that each call to the black box assigns a value of $f(j)$ to some $j$ is at least $1 / 2$. Based on this, use a Chernoff bound to bound the probability that the number of calls to the black box is at least $4 n$.

Victor Salazar

Numerade Educator

View

Problem 9

Suppose that we can obtain independent samples $X_{1}, X_{2}, \ldots$ of a random variable $X$ and that we want to use these samples to estimate $\mathbf{E}[X]$. Using $t$ samples, we use $\left(\sum_{i=1}^{t} X_{i}\right) / t$ for our estimate of $\mathbf{E}[X]$. We want the estimate to be within $\varepsilon \mathbf{E}[X]$ from the true value of $\mathbf{E}[X]$ with probability at least $1-\delta$. We may not be able to use Chernoff's bound directly to bound how good our estimate is if $X$ is not a 0-1 random variable, and we do not know its moment generating function. We develop an alternative approach that requires only having a bound on the variance of $X$. Let $r=\sqrt{\operatorname{Var}[X]} / \mathbf{E}[X]$.
(a) Show using Chebyshev's inequality that $O\left(r^{2} / \varepsilon^{2} \delta\right)$ samples are sufficient to solve the problem.
(b) Suppose that we need only a weak estimate that is within $\varepsilon \mathbf{E}[X]$ of $\mathbf{E}[X]$ with probability at least $3 / 4$. Argue that $O\left(r^{2} / \varepsilon^{2}\right)$ samples are enough for this weak estimate.
(c) Show that, by taking the median of $O(\log (1 / \delta))$ weak estimates, we can obtain an estimate within $\varepsilon \mathbf{E}[X]$ of $\mathbf{E}[X]$ with probability at least $1-\delta$. Conclude that we need only $O\left(\left(r^{2} \log (1 / \delta)\right) / \varepsilon^{2}\right)$ samples.

Victor Salazar

Numerade Educator

01:15

Problem 10

A casino is testing a new class of simple slot machines. Each game, the player puts in $$\$ 1$$, and the slot machine is supposed to return either $$\$ 3$$ to the player with probability $$4 / 25, \$ 100$$ with probability $1 / 200$, or nothing with all remaining probability. Each game is supposed to be independent of other games.

The casino has been surprised to find in testing that the machines have lost $$\$ 10,000$$ over the first million games. Derive a Chernoff bound for the probability of this event. You may want to use a calculator or program to help you choose appropriate values as you derive your bound.

A M

Numerade Educator

02:36

Problem 11

Consider a collection $X_{1}, \ldots, X_{n}$ of $n$ independent integers chosen uniformly from the set $\{0,1,2\}$. Let $X=\sum_{i=1}^{n} X_{i}$ and $0<\delta<1$. Derive a Chernoff bound for $\operatorname{Pr}(X \geq(1+\delta) n)$ and $\operatorname{Pr}(X \leq(1-\delta) n)$.

Amany Waheeb

Numerade Educator

02:36

Problem 12

Consider a collection $X_{1}, \ldots, X_{n}$ of $n$ independent geometrically distributed random variables with mean $2 .$ Let $X=\sum_{i=1}^{n} X_{i}$ and $\delta>0$.
(a) Derive a bound on $\operatorname{Pr}(X \geq(1+\delta)(2 n))$ by applying the Chernoff bound to a sequence of $(1+\delta)(2 n)$ fair coin tosses.
(b) Directly derive a Chernoff bound on $\operatorname{Pr}(X \geq(1+\delta)(2 n))$ using the moment generating function for geometric random variables.
(c) Which bound is better?

Amany Waheeb

Numerade Educator

View

Problem 13

Let $X_{1}, \ldots, X_{n}$ be independent Poisson trials such that $\operatorname{Pr}\left(X_{i}=1\right)=$ $p$. Let $X=\sum_{i=1}^{n} X_{i}$, so that $\mathbf{E}[X]=p n$. Let
$$
F(x, p)=x \ln (x / p)+(1-x) \ln ((1-x) /(1-p)) .
$$
(a) Show that, for $1 \geq x>p$,
$$
\operatorname{Pr}(X \geq x n) \leq \mathrm{e}^{-n F(x, p)}
$$
(b) Show that, when $0<x, p<1$, we have $F(x, p)-2(x-p)^{2} \geq 0$. (Hint: Take the second derivative of $F(x, p)-2(x-p)^{2}$ with respect to $x$.)
(c) Using parts (a) and (b), argue that
$$
\operatorname{Pr}(X \geq(p+\varepsilon) n) \leq \mathrm{e}^{-2 n \varepsilon^{2}}
$$
(d) Use symmetry to argue that
$$
\operatorname{Pr}(X \leq(p-\varepsilon) n) \leq \mathrm{e}^{-2 n \varepsilon^{2}},
$$
and conclude that
$$
\operatorname{Pr}(|X-p n| \geq \varepsilon n) \leq 2 \mathrm{e}^{-2 n \varepsilon^{2}}
$$

Victor Salazar

Numerade Educator

04:40

Problem 14

Modify the proof of Theorem $4.4$ to show the following bound for a weighted sum of Poisson trials. Let $X_{1}, \ldots, X_{n}$ be independent Poisson trials such that $\operatorname{Pr}\left(X_{i}\right)=p_{i}$ and let $a_{1}, \ldots, a_{n}$ be real numbers in $[0,1]$. Let $X=\sum_{i=1}^{n} a_{i} X_{i}$ and $\mu=$ $\mathbf{E}[X]$. Then the following Chernoff bound holds: for any $\delta>0$,
$$
\operatorname{Pr}(X \geq(1+\delta) \mu) \leq\left(\frac{\mathrm{e}^{\delta}}{(1+\delta)^{(1+\delta)}}\right)^{\mu}
$$
Prove a similar bound for the probability that $X \leq(1-\delta) \mu$ for any $0<\delta<1$.

Robin Corrigan

Numerade Educator

02:36

Problem 15

Let $X_{1}, \ldots, X_{n}$ be independent random variables such that
$$
\operatorname{Pr}\left(X_{i}=1-p_{l}\right)=p_{i} \quad \text { and } \operatorname{Pr}\left(X_{i}=-p_{i}\right)=1-p_{i} .
$$
Let $X=\sum_{i=1}^{n} X_{i}$. Prove
$$
\operatorname{Pr}(|X| \geq a) \leq 2 \mathrm{e}^{-2 a^{2} / n}
$$
Hint: You may need to assume the inequality
$$
p_{i} \mathrm{e}^{\lambda\left(1-p_{t}\right)}+\left(1-p_{i}\right) \mathrm{e}^{-\lambda p_{1}} \leq \mathrm{e}^{\lambda^{2} / 8}
$$
This inequality is difficult to prove directly.

Amany Waheeb

Numerade Educator

02:44

Problem 16

Let $X_{1}, \ldots, X_{n}$ be independent Poisson trials such that $\operatorname{Pr}\left(X_{i}\right)=p_{i}$. Let $X=\sum_{i=1}^{n} a_{i} X_{i}$ and $\mu=\mathbf{E}[X]$. Use the result of Exercise $4.15$ to prove that, for any $0<\delta<1$
$$
\operatorname{Pr}(|X-\mu| \geq \delta \mu) \leq 2 \mathrm{e}^{-2 \delta^{2} \mu^{2} / n}
$$

Khoobchandra Agrawal

Numerade Educator

View

Problem 17

Suppose that we have $n$ jobs to distribute among $m$ processors. For simplicity, we assume that $m$ divides $n$. A job takes 1 step with probability $p$ and $k>1$ steps with probability $1-p$. Use Chernoff bounds to determine upper and lower bounds (that hold with high probability) on when all jobs will be completed if we randomly assign exactly $n / m$ jobs to each processor.

Victor Salazar

Numerade Educator

01:22

Problem 18

In many wireless communication systems, each receiver listens on a specific frequency. The bit $b(t)$ sentat time $t$ is represented by a 1 or $-1$. Unfortunately, noise from other nearby communications can affect the receiver's signal. A simplified model of this noise is as follows. There are $n$ other senders, and the $i$ th has strength $p_{i} \leq 1$. At any time $t$, the $i$ th sender is also trying to send a bit $b_{i}(t)$ that is represented by $l$ or $-1$. The receiver obtains the signal $s(t)$ given by
$$
s(t)=b(t)+\sum_{i=1}^{n} p_{i} b_{i}(t)
$$
If $s(t)$ is closer to 1 than $-1$, the receiver assumes that the bit sent at time $t$ was a 1 ; otherwise, the receiver assumes that it was a $-1$.
Assume that all the bits $b_{i}(t)$ can be considered independent, uniform random variables. Give a Chernoff bound to estimate the probability that the receiver makes an error in determining $b(t)$.

Dominador Tan

Numerade Educator

01:59

Problem 19

Recall that a function $f$ is said to be convex if, for any $x_{1}, x_{2}$ and for $0 \leq \lambda \leq 1$,
$$
f\left(\lambda x_{1}+(1-\lambda) x_{2}\right) \leq \lambda f\left(x_{1}\right)+(1-\lambda) f\left(x_{2}\right) .
$$
(a) Let $Z$ be a random variable that takes on a (finite) set of values in the interval $[0,1]$, and let $p=\mathbf{E}[Z]$. Define the Bernoulli random variable $X$ by $\operatorname{Pr}(X=1)=p$ and $\operatorname{Pr}(X=0)=1-p$. Show that $\mathbf{E}[f(Z)] \leq \mathbf{E}[f(X)]$ for any convex function $f$.
(b) Use the fact that $f(x)=\mathrm{e}^{t x}$ is convex for any $t \geq 0$ to obtain a Chernoff-like bound for $Z$ based on a Chernoff bound for $X$.

Minh Le

Numerade Educator

04:00

Problem 20

We prove that the Randomized Quicksort algorithm sorts a set of $n$ numbers in time $O(n \log n)$ with high probability. Consider the following view of Randomized Quicksort. Every point in the algorithm where it decides on a pivot element is called a node. Suppose the size of the set to be sorted at a particular node is $s$. The node is called good if the pivot element divides the set into two parts, each of size not exceeding $2 s / 3$. Otherwise the node is called $b a d$. The nodes can be thought of as forming a tree in which the root node has the whole set to be sorted and its children have the two sets formed after the first pivot step and so on.
(a) Show that the number of good nodes in any path from the root to a leaf in this tree is not greater than $c \log _{2} n$, where $c$ is some positive constant.
(b) Show that, with high probability (greater than $1-1 / n^{2}$ ), the number of nodes in a given root to leaf path of the tree is not greater than $c^{\prime} \log _{2} n$, where $c^{\prime}$ is another constant.
(c) Show that, with high probability (greater than $1-1 / n$ ), the number of nodes in the longest root to leaf path is not greater than $c^{\prime} \log _{2} n$. (Hint: How many nodes are there in the tree?)
(d) Use your answers to show that the running time of Quicksort is $O(n \log n)$ with probability at least $1-1 / n$.

Bobby Barnes

University of North Texas

02:50

Problem 21

Consider the bit-fixing routing algorithm for routing a permutation on the $n$-cube. Suppose that $n$ is even. Write each source node $s$ as the concatenation of two binary strings $a_{s}$ and $b_{s}$ each of length $n / 2 .$ Let the destination of $s$ 's packet be the concatenation of $b_{s}$ and $a_{s}$. Show that this permutation causes the bit-fixing routing algorithm to take $\Omega(\sqrt{N})$ steps.

Clarissa Noh

Numerade Educator

05:01

Problem 22

Consider the following modification to the bit-fixing routing algorithm for routing a permutation on the $n$-cube. Suppose that, instead of fixing the bits in order from 1 to $n$, each packet chooses a random order (independent of other packets' choices) and fixes the bits in that order. Show that there is a permutation for which this algorithm requires $2^{\Omega(n)}$ steps with high probability.

Amany Waheeb

Numerade Educator

01:36

Problem 23

Assume that we use the randomized routing algorithm for the $n$-cube network (Algorithm 4.2) to route a total of up to $p 2^{n}$ packets, where each node is the source of no more than $p$ packets and each node is the destination of no more than $p$ packets.
(a) Give a high-probability bound on the run-time of the algorithm.
(b) Give a high-probability bound on the maximum number of packets at any node at any step of the execution of the routing algorithm.

Breanna Ollech

Numerade Educator

05:52

Problem 24

Show that the expected number of packets that traverse any edge on the path of a given packet when routing a random permutation on the wrapped butterfly network of $N=n 2^{n}$ nodes is $\Omega\left(n^{2}\right)$.

Robin Corrigan

Numerade Educator

18:04

Problem 25

In this exercise, we design a randomized algorithm for the following packet routing problem. We are given a network that is an undirected connected graph $G$, where nodes represent processors and the edges between the nodes represent wires. We are also given a set of $N$ packets to route. For each packet we are given a source node, a destination node, and the exact route (path in the graph) that the packet should take from the source to its destination. (We may assume that there are no loops in the path.) In each time step, at most one packet can traverse an edge. A packet can wait at any node during any time step, and we assume unbounded queue sizes at each node.
A schedule for a set of packets specifies the timing for the movement of packets along their respective routes. That is, it specifies which packet should move and which should wait at each time step. Our goal is to produce a schedule for the packets that tries to minimize the total time and the maximum queue size needed to route all the packets to their destinations.
(a) The dilation $d$ is the maximum distance traveled by any packet. The congestion $c$ is the maximum number of packets that must traverse a single edge during the entire course of the routing. Argue that the time required for any schedule should be at least $\Omega(c+d)$.
(b) Consider the following unconstrained schedule, where many packets may traverse an edge during a single time step. Assign each packet an integral delay chosen randomly, independently, and uniformly from the interval $[1,\lceil\alpha c / \log (N d)\rceil]$, where $\alpha$ is a constant. A packet that is assigned a delay of $x$ waits in its source node for $x$ time steps; then it moves on to its final destination through its specified route without ever stopping. Give an upper bound on the probability that more than $O(\log (N d))$ packets use a particular edge $e$ at a particular time step $t$.
(c) Again using the unconstrained schedule of part (b), show that the probability that more than $O(\log (N d))$ packets pass through any edge at any time step is at most $1 /(N d)$ for a sufficiently large $\alpha$.
(d) Use the unconstrained schedule to devise a simple randomized algorithm that, with high probability, produces a schedule of length $O(c+d \log (N d))$ using queues of size $O(\log (N d))$ and following the constraint that at most one packet crosses an edge per time step.

Ren Jie Tuieng

Numerade Educator