Consider a k -armed bandit problem with k=4 actions, denoted 1,2,3, and 4 . Consider applying to this problem a bandit algorithm using \epsi lon-greedy action
selection, sample-average action-value estimates, and initial estimates of Q1(a) =0, for all a. Suppose the initial sequence of actions and rewards is A_(1)=1,
R_(1)=-1,A_(2)=2,R_(2)=1,A_(3)=2,R_(3)=-2,A_(4)=2,R_(4)=2,A_(5)=3,R_(5)=0. On some of these time steps the \epsi lon case may have occurred, causing an action to be
selected at random. On which time steps could this possibly (including must definitely) have occurred?
