Healthcare: Consider a healthcare system that categorizes the possible overall state of health of the region into three possible states, say i = {1, 2, 3}. Let us say the state is 2 at time zero. The state can transition from state 1 only to state 2 (let p1,2 = 1 and p1,j = 0 if j ≠ 2). Similarly, the state can transition from state 3 only to state 2 (let p3,2 = 1 and let p3,j = 0 if j ≠ 2). However, from state 2, it can transition to state 3 with probability p2,3 > 0 and to state 1 with probability p2,1 = 1 − p2,3 (let p2,2 = 0). Each time it reaches a state i, it spends a random amount of time, say Ti whose distribution is Fi, independent of all other events. There is some staffing decision that the healthcare system has to make. It would like to follow the policy of having ni members on the staff when the state is i. Assume that the cost per unit time that this system incurs is c(n, i) if the state is i and there are n members on the staff. (So, when we reach i, we start incurring costs at the rate of c(ni, i) under our current policy.)
(a) Let μi = E[Ti] and let {πi} be the stationary probabilities of the discrete time Markov Chain (DTMC) whose probability of immediately transitioning from any state i to any state j is given by pij. Find the long run rate at which this system incurs costs using the concept of Semi-Markov processes in Ross (2014).
(b) For the DTMC mentioned above, solve for the stationary probabilities {πi}. Substitute these probabilities into the answer in (a). Let this long run rate be θ.
(c) Let us now assume that there is a lead time of k time units involved before a change in the staffing can be implemented. That is, if the state changes from 1 to 2 now, the staffing level will change from n1 to n2 only k time units from now. Assume Ti ≥ k with probability 1 for all i; thus, the system will not change its state within k time units. So, when we reach state i from state j, we incur costs at the rate of c(nj, i) for the first k time units and costs at the rate of c(ni, i) subsequently until we leave state i. Let the long run rate at which this system incurs costs be denoted by θ̂. Derive a simple expression for θ̂ − θ. Hint: Study the regenerative process defined by the entry of the system into state 2. (We could have also defined a regenerative process based on entry into state 1 or based on entry into state 3. However, state 2 entries offer a much simpler analysis.)