• Home
  • Textbooks
  • Probability and Computing
  • Markov Chains and Random Walks

Probability and Computing

Michael Mitzenmacher , Eli Upfal

Chapter 7

Markov Chains and Random Walks - all with Video Answers

Educators

Chapter Questions

01:51

Problem 1

Consider a Markov chain with state space $\{0,1,2,3\}$ and a transition matrix
so $P_{0,3}=3 / 5$ is the probability of moving from state 0 to state
$=\left[\begin{array}{cccc}0 & 3 / 10 & 1 / 10 & 3 / 5 \\ 1 / 10 & 1 / 10 & 7 / 10 & 1 / 10 \\ 1 / 10 & 7 / 10 & 1 / 10 & 1 / 10 \\ 9 / 10 & 1 / 10 & 0 & 0\end{array}\right]$ ility of moving from state 0 to state 3 .
so $P_{0,3}=3 / 5$ is the probability of moving from state 0 to
(b) Find the probability of being in state 3 after 32 steps if the chain begins at state 0 .
(c) Find the probability of being in state 3 after 128 steps if the chain begins at a state chosen uniformly at random from the four states.
(d) Suppose that the chain begins in state 0 . What is the smallest value of $t$ for which $\max _{s}\left|P_{0, s}^{t}-\pi_{*}\right| \leq 0.01 ?$ Here $\tilde{\pi}$ is the stationary distribution. What is the smallest value of $t$ for which $\max _{3}\left|P_{0}^{2}-\pi_{2}\right|<0,001$ ?

Dominador Tan
Dominador Tan
Numerade Educator
07:59

Problem 2

Consider the two-state Markov chain with the following transition matrix.
$$
\mathbf{P}=\left[\begin{array}{cc}
p & 1-p \\
1-p & p
\end{array}\right]
$$
Find a simple expression for $P_{0,0^{\circ}}^{t}$

Mengchun Cai
Mengchun Cai
Numerade Educator
04:10

Problem 3

Consider a process $X_{0}, X_{1}, X_{2}, \ldots$ with two states, 0 and 1 . The process is governed by two matrices, $\mathbf{P}$ and $\mathbf{Q}$. If $k$ is even, the values $P_{t, j}$ give the probability of going from state $i$ to state $j$ on the step from $X_{k}$ to $X_{k+1}$. Likewise, if $k$ is odd then the values $Q_{i, j}$ give the probability of going from state $i$ to state $j$ on the step from $X_{k}$ to $X_{k+1}$. Explain why this process does not satisfy Definition 7.1 of a (time-homogeneous) Markov chain. Then give a process with a larger state space that is equivalent to this process and satisfies Definition 7.1.

Mahnoor Amin
Mahnoor Amin
Numerade Educator
03:34

Problem 4

Prove that the communicating relation defines an equivalence relation.

Diwakar Mandilwar
Diwakar Mandilwar
Numerade Educator
05:33

Problem 5

Prove that if one state in a communicating class is transient (respectively, recurrent) then all states in that class are transient (respectively, recurrent).

Chris Trentman
Chris Trentman
Numerade Educator
01:39

Problem 6

In studying the 2-SAT algorithm, we considered a 1-dimensional random walk with a completely reflecting boundary at 0 . That is, whenever position 0 is reached, with probability 1 the walk moves to position 1 at the next step. Consider now a random walk with a partially reflecting boundary at 0 . Whenever position 0 is reached, with probability $1 / 2$ the walk moves to position 1 and with probability $1 / 2$ the walk stays at 0 . Everywhere else the random walk moves either up or down 1 , each with probability $1 / 2$. Find the expected number of moves to reach $n$, starting from position $i$ and using a random walk with a partially reflecting boundary.

Dominador Tan
Dominador Tan
Numerade Educator
01:25

Problem 7

Suppose that the 2-SAT Algorithm $7.1$ starts with an assignment chosen uniformly at random. How does this affect the expected time until a satisfying assignment is found?

Nick Johnson
Nick Johnson
Numerade Educator
01:25

Problem 8

Generalize the randomized algorithm for 3-SAT to $k-S A T$. What is the expected time of the algorithm as a function of $k$ ?

Nick Johnson
Nick Johnson
Numerade Educator
View

Problem 9

In the analysis of the randomized algorithm for 3-SAT, we made the pessimistic assumption that the current assignment $A_{i}$ and the truth assignment $S$ differ on just one variable in the clause chosen at each step. Suppose instead that, independently at each step, the two assignments disagree on one variable in the clause with probability $p$ and at least two variables with probability $1-p$. What is the largest value of $p$ for which you can prove that the expected number of steps before Algorithm $7.2$ terminates is polynomial in $p$ ? Give a proof for this value of $p$ and give an upper bound on the expected number of steps in this case.

Shu Naito
Shu Naito
Numerade Educator
05:12

Problem 10

A coloring of a graph is an assignment of a color to each of its vertices. A graph is $k$-colorable if there is a coloring of the graph with $k$ colors such that no two adjacent vertices have the same color. Let $G$ be a 3 -colorable graph.
(a) Show that there exists a coloring of the graph with two colors such that no triangle is monochromatic. (A triangle of a graph $G$ is a subgraph of $G$ with three vertices, which are all adjacent to each other.)
(b) Consider the following algorithm for coloring the vertices of $G$ with two colors so that no triangle is monochromatic. The algorithm begins with an arbitrary 2coloring of $G$. While there are any monochromatic triangles in $G$, the algorithm chooses one such triangle and changes the color of a randomly chosen vertex of that triangle. Derive an upper bound on the expected number of such recoloring steps before the algorithm finds a 2-coloring with the desired property.

Carson Merrill
Carson Merrill
Numerade Educator
View

Problem 11

An $n \times n$ matrix P with entries $P_{i, j}$ is called stochastic if all entries are nonnegative and if the sum of the entries in each row is 1 . It is called doubly stochastic if, additionally, the sum of the entries in each column is 1 . Show that the uniform distribution is a stationary distribution for any Markov chain represented by a donbly stochastic matrix.

Aishwarya Krishnakumar
Aishwarya Krishnakumar
Numerade Educator
04:41

Problem 12

Let $X_{n}$ be the sum of $n$ independent rolls of a fair die. Show that, for any $k \geq 2$
$$
\lim _{n \rightarrow \infty} \operatorname{Pr}\left(X_{n} \text { is divisible by } k\right)=\frac{1}{k}
$$

Mengchun Cai
Mengchun Cai
Numerade Educator
01:51

Problem 13

Consider a finite Markov chain on $n$ states with stationary distribution $\bar{\pi}$ and transition probabilities $P_{i, j}$. Imagine starting the chain at time 0 and running it for $m$ steps, obtaining the sequence of states $X_{0}, X_{1, \ldots .,} X_{m}$. Consider the states in reverse order, $X_{m}, X_{m-1}, \ldots, X_{0}$.
(a) Argue that given $X_{k+1}$, the state $X_{k}$ is independent of $X_{k+2}, X_{k+3}, \ldots, X_{m} .$ Thus the reverse sequence is Markovian.
(b) Argue that for the reverse sequence, the transition probabilities $Q_{i, j}$ are given by
$$
Q_{i, j}=\frac{\pi_{j} P_{j, i}}{\pi_{i}}
$$
(c) Prove that if the original Markov chain is time reversible, so that $\pi_{i} P_{i, j}=\pi_{j} P_{j, i}$ then $Q_{i, j}=P_{i, j}$. That is, the states follow the same transition probabilities whether viewed in forward order or reverse order.

Dominador Tan
Dominador Tan
Numerade Educator
04:37

Problem 14

Prove that the Markov chain corresponding to a random walk on an undirected, non-bipartite graph that consists of one component is time reversible.

Robin Corrigan
Robin Corrigan
Numerade Educator
01:54

Problem 15

Let $P_{i, i}^{t}$ be the probability that a Markov chain returns to state $i$ when started in state $i$ after $t$ steps. Prove that
$$
\sum_{i=1}^{\infty} P_{i, i}^{t}
$$
is unbounded if and only if state $i$ is recurrent.

Nick Johnson
Nick Johnson
Numerade Educator
08:18

Problem 16

Prove Lemma $7.5 .$

Chris Trentman
Chris Trentman
Numerade Educator
01:39

Problem 17

Consider the following Markov chain, which is similar to the 1-dimensional random walk with a completely reflecting boundary at 0 . Whenever position 0 is reached, with probability 1 the walk moves to position 1 at the next step. Otherwise, the walk moves from $i$ to $i+1$ with probability $p$ and from $i$ to $i-1$ with probability $1-p$. Prove that:
(a) if $p<1 / 2$, each state is positive recurrent;
(b) if $p=1 / 2$, each state is null recurrent;
(c) if $p>1 / 2$, each state is transient.

Dominador Tan
Dominador Tan
Numerade Educator
01:39

Problem 18

(a) Consider a random walk on the 2-dimensional integer lattice, where each point has four neighbors (up, down, left, and right). Is each state transient, null recurrent, or positive recurrent? Give an argument,
(b) Answer the problem in (a) for the 3-dimensional integer lattice.

Dominador Tan
Dominador Tan
Numerade Educator
03:31

Problem 19

Consider the gambler's ruin problem, where a player plays until they lose $\ell_{1}$ dollars or win $\ell_{2}$ dollars. Prove that the expected number of games played is $\ell_{1} \ell_{2}$.

Nick Johnson
Nick Johnson
Numerade Educator
03:31

Problem 20

We have considered the gambler's ruin problem in the case where the game is fair. Consider the case where the game is not fair; instead, the probability of losing a dollar each game is $2 / 3$ and the probability of winning a dollar each game is 1/3. Suppose that you start with $i$ dollars and finish either when you reach $n$ or lose it all. Let $W_{l}$ be the amount you have gained after $t$ rounds of play.
(a) Show that $\mathbf{E}\left[2^{W_{l+1}}\right]=\mathbf{E}\left[2^{W_{I}}\right]$.
(b) Use part (a) to determine the probability of finishing with 0 dollars and the probability of finishing with $n$ dollars when starting at position $i$.
(c) Generalize the preceding argument to the case where the probability of losing is $p>1 / 2 .$ (Hint: Try considering $\mathbf{E}\left[c^{W_{r}}\right]$ for some constant $c_{.}$)

Nick Johnson
Nick Johnson
Numerade Educator
01:39

Problem 21

Consider a Markov chain on the states $\{0,1, \ldots, n\}$, where for $i<n$ we have $P_{i, i+1}=1 / 2$ and $P_{i, 0}=1 / 2$. Also, $P_{n, n}=1 / 2$ and $P_{n, 0}=1 / 2$. This process can be viewed as a random walk on a directed graph with vertices $\{0,1, \ldots, n\}$, where each vertex has two directed edges: one that returns to 0 and one that moves to the vertex with the next higher number (with a self-loop at vertex $n$ ). Find the stationary distribution of this chain. (This example shows that random walks on directed graphs are very different than random walks on undirected graphs.)

Dominador Tan
Dominador Tan
Numerade Educator
04:21

Problem 22

A cat and a mouse each independently take a random walk on a connected, undirected, non-bipartite graph $G$. They start at the same time on different nodes, and each makes one transition at each time step. The cat eats the mouse if they are ever at the same node at some time step. Let $n$ and $m$ denote, respectively, the number of vertices and edges of $G .$ Show an upper bound of $O\left(m^{2} n\right)$ on the expected time before the cat eats the mouse. (Hint: Consider a Markov chain whose states are the ordered pairs $(a, b)$, where $a$ is the position of the cat and $b$ is the position of the mouse.)

Dominador Tan
Dominador Tan
Numerade Educator
02:31

Problem 23

One way of spreading information on a network uses a rumor-spreading paradigm. Suppose that there are $n$ hosts currently on the network. Initially, one host begins with a message. Each round, every host that has the message contacts another host chosen independently and uniformly at random from the other $n-1$ hosts, and sends that host the message. We would like to know how many rounds are necessary before all hosts have received the message with probability $0.99$.
(a) Explain how this problem can be viewed in terms of Markov chains.
(b) Determine a method for computing the probability that $j$ hosts have received the message after round $k$ given that $i$ hosts have received the message after round $k-1$. (Hint: There are various ways of doing this. One approach is to let $P(i, j, c)$ be the probability that $j$ hosts have the message after the first $c$ of the $i$ hosts have made their choices in a round; then find a recurrence for $P$.)
(c) As a computational exercise, write a program to determine the number of rounds required for a message starting at one host to reach all other hosts with probability $0.9999$ when $n=128$

Nick Johnson
Nick Johnson
Numerade Educator
03:13

Problem 24

The lollipop graph on $n$ vertices is a clique on $n / 2$ vertices connected to a path on $n / 2$ vertices, as shown in Figure 7.3. The node $u$ is a part of both the clique and the path. Let $v$ denote the other end of the path.
185 (a) Show that the expected covering time of a random walk starting at $v$ is $\Theta\left(n^{2}\right)$.
(b) Show that the expected covering time for a random walk starting at $u$ is $\Theta\left(n^{3}\right)$.

Amany Waheeb
Amany Waheeb
Numerade Educator
03:34

Problem 26

Let $n$ equidistant points be marked on a circle. Without loss of generality, we think of the points as being labeled clockwise from 0 to $n-1$. Initially, a wolf begins at 0 and there is one sheep at each of the remaining $n-1$ points. The wolf takes a random walk on the circle. For each step, it moves with probability $1 / 2$ to one neighboring point and with probability $1 / 2$ to the other neighboring point. At the first visit to a point, the wolf eats a sheep if there is still one there. Which sheep is most likely to be the last eaten?

Amany Waheeb
Amany Waheeb
Numerade Educator
01:51

Problem 27

Suppose that we are given $n$ records, $R_{1}, R_{2}, \ldots, R_{n} .$ The records are kept in some order. The cost of accessing the $j$ th record in the order is $j$. Thus, if we had four records ordered as $R_{2}, R_{4}, R_{3}, R_{1}$, then the cost of accessing $R_{4}$ would be 2 and the cost of accessing $R_{1}$ would be 4 .

Suppose further that, at each step, record $R_{j}$ is accessed with probability $p_{j}$, with each step being independent of other steps. If we knew the values of the $p_{j}$ in advance, we would keep the $R_{j}$ in decreasing order with respect to $p_{j}$. But if we don't know the $p_{j}$ in advance, we might use the "move to front" heuristic: at each step, put the record that was accessed at the front of the list. We assume that moving the record can be done with no cost and that all other records remain in the same order. For example, if the order was $R_{2}, R_{4}, R_{3}, R_{1}$ before $R_{3}$ was accessed, then the order at the next step would be $R_{3}, R_{2}, R_{4}, R_{1}$.

In this setting, the order of the records can be thought of as the state of a Markov chain. Give the stationary distribution of this chain. Also, let $X_{k}$ be the cost for accessing the $k$ th requested record. Determine an expression for $\lim _{k \rightarrow \infty} \mathbf{E}\left[X_{k}\right]$. Your expression should be easily computable in time that is polynomial in $n$, given the $p_{\jmath}$.

Dominador Tan
Dominador Tan
Numerade Educator
06:56

Problem 28

Consider the following variation of the discrete time queue. Time is divided into fixed-length steps. At the beginning of each time step, a customer arrives with probability $\lambda$. At the end of each time step, if the queue is nonempty then the customer at the front of the line completes service with probability $\mu$.
(a) Explain how the number of customers in the queue at the beginning of each time step forms a Markov chain, and determine the corresponding transition probabilities.
(b) Explain under what conditions you would expect a stationary distribution $\bar{\pi}$ to exist.
(c) If a stationary distribution exists, then what should be the value of $\pi_{0}$, the probability that no customers are in the queue at the beginning of the time step? (Hint: Consider that, in the long ron, the rate at which customers enter the queue and the rate at which customers leave the queue must be equal.)
(d) Determine the stationary distribution and explain how it corresponds to your conditions from part (b).
(e) Now consider the variation where we change the order of incoming arrivals and service. That is: at the beginning of each time step, if the queue is nonempty then a customer is served with probability $\mu$; and at the end of a time step a customer arrives with probability $\lambda$. How does this change your answers to parts (a)-(d)?

Robin Corrigan
Robin Corrigan
Numerade Educator