Question 10 (18 points) Consider a Markov chain with states 1,2,3 having transition
probability matrix
$\begin{pmatrix}
0.5 & 0.3 & 0.2 \\
0 & 0.4 & 0.6 \\
0.8 & 0 & 0.2
\end{pmatrix}$
3
(a) (2 points) If the chain is currently in state 1, find the probability that after two
transitions it will be in state 2.
(b) (4 points) Suppose you receive a reward $r(i) = i^2$ whenever the Markov chain is
in state $i$, $i = 1, 2, 3$. Find your long run average reward.
