Consider the following problem: Let X = {B, G}, U = {0, 1}, where X denotes whether a fading channel is in a good state (G) or a bad state (B). There exists an encoder who can either try to use the channel (u = 1) or not use the channel (u = 0). The goal of the encoder is to send information across the channel.
Suppose that the encoder's per-stage cost (to be minimized) is given by:
c(x, u=1) = {r=G, u=1} + u, for some (, n R to be specified below.
If you view this as a maximization problem, you can see that the goal is to maximize information transmission efficiency subject to a cost involving an attempt to use the channel. The model can be made more complicated, but the idea is that when the channel state is good, u = 1 can represent a channel input which contains data to be transmitted, and u = 0 denotes that the channel is not used. When u = 1 and = G, the channel is utilized successfully.
For many channels with memory, the input also impacts the channel state. Suppose that the transition kernel is given by:
P(x+1=G|x=G, u=1) = 0.1,
P(x+1=B|x=G, =1) = 0.9,
P(x+1=G|x=G, =0) = 0.9,
P(x+1=B|x=G, u=0) = 0.1,
P(x+1=G|x=B, u=1) = 0.5,
P(x+1=G|x=B, u=0) = 0.9,
P(x+1=B|x=B, u=1) = 0.5,
P(x+1=B|x=B, u=0) = 0.1.
We will consider either a discounted cost criterion for some (0,1) (you can fix an arbitrary value):
infE[c(x,u)] YETA t=0 (1),
or the average cost criterion:
T
r T
t=0.
a) Using Matlab or some other program, obtain a solution to the problem given above in (1) through the following:
(i) [15 Points] Value Iteration. Take some fixed (0, 1) of your choice. Consider = 0.75 and n = 0.8, n = 0.6, and n = 0.01. Interpret the optimal solution for these different values of n, in view of the application.
(ii) [15 Points] Policy Iteration. With the same as above, and = 0.75, work again with each of the following: n = 0.8, n = 0.6, and n = 0.01.
(iii) [20 Points] Q-Learning. With the same as above, try only n = 0.6. Note that a common way to pick coefficients in the Q-learning algorithm is to take for every (, u) pair: 1 ax,u = 1 + D = 0.1{x=2,u=u}. Compare your solutions (obtained via the different methods).
b) [20 Points] Consider the criterion given in (2). Apply the convex analytic method, by solving the corresponding linear program, to find the optimal policy. Take = 0.75 and n = 0.6. In Matlab, the command linproq can be used to solve linear programming problems. See the lecture notes.
