4. Markov Game
Your friend and you play a board game. Each takes turn to play one of the two possible moves. Let call
them L and R. Each of you comes up with your own strategy based on the immediate previous move of your
oponent. Specifically, you will use the same move as your friend's move with probability $p$ and $1 - p$ otherwise.
For example, if your friend's last move is L, you will move L with probability $p$ and move R with probability
$1 - p$. Similarly, your friend will use the same move as yours with probability $q$ and $1 - q$ otherwise. Let
$X_1, X_2, X_3,...$, denote a sequence of moves from you and your friend, $X_i \in \{L, R\}$.
(a) Show that $X_1, X_2, X_3,...$ is not a stationary random process.
(b) Show that the sequence $X_1, X_3, X_5,...$ is a stationary process. Compute the stationary distribution of
$X_1, X_3, X_5,...$
(c) Compute the entropy rate $R$ of the sequence $X_1, X_3, X_5,...$
(d) Repeat questions (b) and (c) for the sequence $X_2, X_4, X_6,...$
(e) Even though strictly speaking, the entropy rate $H(X) = X_1, X_2, X_3,...$ does not exist, can you come up
with your own entropy rate for this scenario by looking at the entropy rate of the odd and even sequences
in part (b) and (c)
(f) Suppose you want to produce the sequence of moves of the game that when compressed has smallest
expected length per move. You cannot control your friend's move, but you can control yours. Suggest a
strategy for doing so. What is the expected length in term of $q$?
