• Home
  • Textbooks
  • Introduction to Probability Models
  • Continuous-Time Markov Chains

Introduction to Probability Models

Sheldon M. Ross

Chapter 6

Continuous-Time Markov Chains - all with Video Answers

Educators


Chapter Questions

04:11

Problem 1

A population of organisms consists of both male and female members. In a small colony any particular male is likely to mate with any particular female in any time interval of length $h$, with probability $\lambda h+o(h)$. Each mating immediately produces one offspring, equally likely to be male or female. Let $N_1(t)$ and $N_2(t)$ denote the number of males and females in the population at $t$. Derive the parameters of the continuous-time Markov chain $\left\{N_1(t), N_2(t)\right\}$, i.e., the $v_i, P_{i j}$ of Section 6.2 .

James Kiss
James Kiss
Numerade Educator
10:03

Problem 2

Suppose that a one-celled organism can be in one of two states-either $A$ or $B$. An individual in state $A$ will change to state $B$ at an exponential rate $\alpha$; an individual in state $B$ divides into two new individuals of type $A$ at an exponential rate $\beta$. Define an appropriate continuous-time Markov chain for a population of such organisms and determine the appropriate parameters for this model.

Robin Corrigan
Robin Corrigan
Numerade Educator

Problem 3

Consider two machines that are maintained by a single repairman. Machine $i$ functions for an exponential time with rate $\mu_i$ before breaking down, $i=1,2$. The repair times (for either machine) are exponential with rate $\mu$. Can we analyze this as a birth and death process? If so, what are the parameters? If not, how can we analyze it?

Check back soon!

Problem 4

Potential customers arrive at a single-server station in accordance with a Poisson process with rate $\lambda$. However, if the arrival finds $n$ customers already in the station, then he will enter the system with probability $\alpha_n$. Assuming an exponential service rate $\mu$, set this up as a birth and death process and determine the birth and death rates.

Check back soon!
04:11

Problem 5

There are $N$ individuals in a population, some of whom have a certain infection that spreads as follows. Contacts between two members of this population occur in accordance with a Poisson process having rate $\lambda$. When a contact occurs, it is equally likely to involve any of the $\left(\begin{array}{c}N \\ 2\end{array}\right)$ pairs of individuals in the population. If a contact involves an infected and a noninfected individual, then with probability $p$ the noninfected individual becomes infected. Once infected, an individual remains infected throughout. Let $X(t)$ denote the number of infected members of the population at time $t$.
(a) Is $\{X(t), t \geqslant 0\}$ a continuous-time Markov chain?
(b) Specify its type.
(c) Starting with a single infected individual, what is the expected time until all members are infected?

James Kiss
James Kiss
Numerade Educator

Problem 6

Consider a birth and death process with birth rates $\lambda_i=(i+1) \lambda, i \geqslant 0$, and death rates $\mu_i=i \mu, i \geqslant 0$.
(a) Determine the expected time to go from state 0 to state 4 .
(b) Determine the expected time to go from state 2 to state 5 .
(c) Determine the variances in parts (a) and (b).

Check back soon!

Problem 7

Individuals join a club in accordance with a Poisson process with rate $\lambda$. Each new member must pass through $k$ consecutive stages to become a full member of the club. The time it takes to pass through each stage is exponentially distributed with rate $\mu$. Let $N_i(t)$ denote the number of club members at time $t$ who have passed through exactly $i$ stages, $i=1, \ldots, k-1$. Also, let $\mathbf{N}(t)=\left(N_1(t), N_2(t), \ldots, N_{k-1}(t)\right)$
(a) Is $\{\mathbf{N}(t), t \geqslant 0\}$ a continuous-time Markov chain?
(b) If so, give the infinitesimal transition rates. That is, for any state $\mathbf{n}=$ $\left(n_1, \ldots, n_{k-1}\right)$ give the possible next states along with their infinitesimal rates.

Check back soon!

Problem 8

Consider two machines, both of which have an exponential lifetime with mean $1 / \lambda$. There is a single repairman that can service machines at an exponential rate $\mu$. Set up the Kolmogorov backward equations; you need not solve them.

Check back soon!

Problem 9

The birth and death process with parameters $\lambda_n=0$ and $\mu_n=\mu, n>0$ is called a pure death process. Find $P_{i j}(t)$.

Check back soon!

Problem 10

Consider two machines. Machine $i$ operates for an exponential time with rate $\lambda_i$ and then fails; its repair time is exponential with rate $\mu_i, i=1,2$. The machines act independently of each other. Define a four-state continuous-time Markov chain which jointly describes the condition of the two machines. Use the assumed independence to compute the transition probabilities for this chain and then verify that these transition probabilities satisfy the forward and backward equations.

Check back soon!
View

Problem 11

Consider a Yule process starting with a single individual-that is, suppose $X(0)=1$. Let $T_i$ denote the time it takes the process to go from a population of size $i$ to one of size $i+1$.
(a) Argue that $T_i, i=1, \ldots, j$, are independent exponentials with respective rates $i \lambda$.
(b) Let $X_1, \ldots, X_j$ denote independent exponential random variables each having rate $\lambda$, and interpret $X_i$ as the lifetime of component $i$. Argue that
$\max \left(X_1, \ldots, X_j\right)$ can be expressed as
$$
\max \left(X_1, \ldots, X_j\right)=\varepsilon_1+\varepsilon_2+\cdots+\varepsilon_j
$$
where $\varepsilon_1, \varepsilon_2, \ldots, \varepsilon_j$ are independent exponentials with respective rates $j \lambda$, $(j-1) \lambda, \ldots, \lambda$
Hint: Interpret $\varepsilon_i$ as the time between the $i-1$ and the $i$ th failure.
(c) Using (a) and (b) argue that
$$
P\left\{T_1+\cdots+T_j \leqslant t\right\}=\left(1-e^{-\lambda t}\right)^j
$$
(d) Use (c) to obtain that
$$
P_{1 j}(t)=\left(1-e^{-\lambda t}\right)^{j-1}-\left(1-e^{-\lambda t}\right)^j=e^{-\lambda t}\left(1-e^{-\lambda t}\right)^{j-1}
$$
and hence, given $X(0)=1, X(t)$ has a geometric distribution with parameter $p=e^{-\lambda t}$.
(e) Now conclude that
$$
P_{i j}(t)=\left(\begin{array}{l}
j-1 \\
i-1
\end{array}\right) e^{-\lambda t i}\left(1-e^{-\lambda t}\right)^{j-i}
$$

Victor Salazar
Victor Salazar
Numerade Educator
01:03

Problem 12

Each individual in a biological population is assumed to give birth at an exponential rate $\lambda$, and to die at an exponential rate $\mu$. In addition, there is an exponential rate of increase $\theta$ due to immigration. However, immigration is not allowed when the population size is $N$ or larger.
(a) Set this up as a birth and death model.
(b) If $N=3,1=\theta=\lambda, \mu=2$, determine the proportion of time that immigration is restricted.

Dominador Tan
Dominador Tan
Numerade Educator

Problem 13

A small barbershop, operated by a single barber, has room for at most two customers. Potential customers arrive at a Poisson rate of three per hour, and the successive service times are independent exponential random variables with mean $\frac{1}{4}$ hour. What is
(a) the average number of customers in the shop?
(b) the proportion of potential customers that enter the shop?
(c) If the barber could work twice as fast, how much more business would he do?

Check back soon!

Problem 14

Potential customers arrive at a full-service, one-pump gas station at a Poisson rate of 20 cars per hour. However, customers will only enter the station for gas if there are no more than two cars (including the one currently being attended to) at the pump. Suppose the amount of time required to service a car is exponentially distributed with a mean of five minutes.
(a) What fraction of the attendant's time will be spent servicing cars?
(b) What fraction of potential customers are lost?

Check back soon!

Problem 15

A service center consists of two servers, each working at an exponential rate of two services per hour. If customers arrive at a Poisson rate of three per hour, then, assuming a system capacity of at most three customers,
(a) what fraction of potential customers enter the system?
(b) what would the value of part (a) be if there was only a single server, and his rate was twice as fast (that is, $\mu=4$ )?

Check back soon!
06:00

Problem 16

The following problem arises in molecular biology. The surface of a bacterium consists of several sites at which foreign molecules-some acceptable and some not-become attached. We consider a particular site and assume that molecules arrive at the site according to a Poisson process with parameter $\lambda$. Among these molecules a proportion $\alpha$ is acceptable. Unacceptable molecules stay at the site for a length of time which is exponentially distributed with parameter $\mu_1$, whereas an acceptable molecule remains at the site for an exponential time with rate $\mu_2$. An arriving molecule will become attached only if the site is free of other molecules. What percentage of time is the site occupied with an acceptable (unacceptable) molecule?

Charles Kirschbaum
Charles Kirschbaum
Numerade Educator

Problem 17

Each time a machine is repaired it remains up for an exponentially distributed time with rate $\lambda$. It then fails, and its failure is either of two types. If it is a type 1 failure, then the time to repair the machine is exponential with rate $\mu_1$; if it is a type 2 failure, then the repair time is exponential with rate $\mu_2$. Each failure is, independently of the time it took the machine to fail, a type 1 failure with probability $p$ and a type 2 failure with probability $1-p$. What proportion of time is the machine down due to a type 1 failure? What proportion of time is it down due to a type 2 failure? What proportion of time is it up?

Check back soon!

Problem 18

After being repaired, a machine functions for an exponential time with rate $\lambda$ and then fails. Upon failure, a repair process begins. The repair process proceeds sequentially through $k$ distinct phases. First a phase 1 repair must be performed, then a phase 2 , and so on. The times to complete these phases are independent, with phase $i$ taking an exponential time with rate $\mu_i, i=1, \ldots, k$.
(a) What proportion of time is the machine undergoing a phase $i$ repair?
(b) What proportion of time is the machine working?

Check back soon!

Problem 19

A single repairperson looks after both machines 1 and 2 . Each time it is repaired, machine $i$ stays up for an exponential time with rate $\lambda_i, i=1,2$. When machine $i$ fails, it requires an exponentially distributed amount of work with rate $\mu_i$ to complete its repair. The repairperson will always service machine 1 when it is down. For instance, if machine 1 fails while 2 is being repaired, then the repairperson will immediately stop work on machine 2 and start on 1 . What proportion of time is machine 2 down?

Check back soon!

Problem 20

There are two machines, one of which is used as a spare. A working machine will function for an exponential time with rate $\lambda$ and will then fail. Upon failure, it is immediately replaced by the other machine if that one is in working order, and it goes to the repair facility. The repair facility consists of a single person who takes an exponential time with rate $\mu$ to repair a failed machine. At the repair facility, the newly failed machine enters service if the repairperson is free. If the repairperson is busy, it waits until the other machine is fixed; at that time, the newly repaired machine is put in service and repair begins on the other one. Starting with both machines in working condition, find
(a) the expected value and
(b) the variance
of the time until both are in the repair facility.
(c) In the long run, what proportion of time is there a working machine?

Check back soon!

Problem 21

Suppose that when both machines are down in Exercise 20 a second repairperson is called in to work on the newly failed one. Suppose all repair times remain exponential with rate $\mu$. Now find the proportion of time at least one machine is working, and compare your answer with the one obtained in Exercise 20.

Check back soon!

Problem 22

Customers arrive at a single-server queue in accordance with a Poisson process having rate $\lambda$. However, an arrival that finds $n$ customers already in the system will only join the system with probability $1 /(n+1)$. That is, with probability $n /(n+1)$ such an arrival will not join the system. Show that the limiting distribution of the number of customers in the system is Poisson with mean $\lambda / \mu$.

Check back soon!
02:12

Problem 23

A job shop consists of three machines and two repairmen. The amount of time a machine works before breaking down is exponentially distributed with mean 10. If the amount of time it takes a single repairman to fix a machine is exponentially distributed with mean 8 , then
(a) what is the average number of machines not in use?
(b) what proportion of time are both repairmen busy?

Hunza Gilgit
Hunza Gilgit
Numerade Educator
View

Problem 24

Consider a taxi station where taxis and customers arrive in accordance with Poisson processes with respective rates of one and two per minute. A taxi will wait no matter how many other taxis are present. However, an arriving customer that does not find a taxi waiting leaves. Find
(a) the average number of taxis waiting, and
(b) the proportion of arriving customers that get taxis.

Victor Salazar
Victor Salazar
Numerade Educator

Problem 25

Customers arrive at a service station, manned by a single server who serves at an exponential rate $\mu_1$, at a Poisson rate $\lambda$. After completion of service the customer then joins a second system where the server serves at an exponential rate $\mu_2$. Such a system is called a tandem or sequential queueing system. Assuming that $\lambda<\mu_i, i=1,2$, determine the limiting probabilities.
Hint: Try a solution of the form $P_{n, m}=C \alpha^n \beta^m$, and determine $C, \alpha, \beta$.

Check back soon!

Problem 26

Consider an ergodic $M / M / s$ queue in steady state (that is, after a long time) and argue that the number presently in the system is independent of the sequence of past departure times. That is, for instance, knowing that there have been departures $2,3,5$, and 10 time units ago does not affect the distribution of the number presently in the system.

Check back soon!

Problem 27

In the $M / M / s$ queue if you allow the service rate to depend on the number in the system (but in such a way so that it is ergodic), what can you say about the output process? What can you say when the service rate $\mu$ remains unchanged but $\lambda>s \mu ?$

Check back soon!

Problem 28

If $\{X(t)\}$ and $\{Y(t)\}$ are independent continuous-time Markov chains, both of which are time reversible, show that the process $\{X(t), Y(t)\}$ is also a time reversible Markov chain.

Check back soon!

Problem 29

Consider a set of $n$ machines and a single repair facility to service these machines. Suppose that when machine $i, i=1, \ldots, n$, fails it requires an exponentially distributed amount of work with rate $\mu_i$ to repair it. The repair facility divides its efforts equally among all failed machines in the sense that whenever there are $k$ failed machines each one receives work at a rate of $1 / k$ per unit time. If there are a total of $r$ working machines, including machine $i$, then $i$ fails at an instantaneous rate $\lambda_i / r$.
(a) Define an appropriate state space so as to be able to analyze the preceding system as a continuous-time Markov chain.
(b) Give the instantaneous transition rates (that is, give the $q_{i j}$ ).
(c) Write the time reversibility equations.
(d) Find the limiting probabilities and show that the process is time reversible.

Check back soon!

Problem 30

Consider a graph with nodes $1,2, \ldots, n$ and the $\left(\begin{array}{c}n \\ 2\end{array}\right) \operatorname{arcs}(i, j), i \neq j$, $i, j,=1, \ldots, n$. (See Section 3.6 .2 for appropriate definitions.) Suppose that a particle moves along this graph as follows: Events occur along the arcs $(i, j)$ according to independent Poisson processes with rates $\lambda_{i j}$. An event along arc $(i, j)$ causes that arc to become excited. If the particle is at node $i$ at the moment that $\left(i, j\right.$ ) becomes excited, it instantaneously moves to node $j, i, j=1, \ldots, n$. Let $P_j$ denote the proportion of time that the particle is at node $j$. Show that
$$
P_j=\frac{1}{n}
$$

Check back soon!

Problem 31

A total of $N$ customers move about among $r$ servers in the following manner. When a customer is served by server $i$, he then goes over to server $j, j \neq i$, with probability $1 /(r-1)$. If the server he goes to is free, then the customer enters service; otherwise he joins the queue. The service times are all independent, with the service times at server $i$ being exponential with rate $\mu, i=1, \ldots, r$. Let the state at any time be the vector $\left(n_1, \ldots, n_r\right)$, where $n_i$ is the number of customers presently at server $i, i=1, \ldots, r, \sum_i n_i=N$.
(a) Argue that if $X(t)$ is the state at time $t$, then $\{X(t), t \geqslant 0\}$, is a continuoustime Markov chain.
(b) Give the infinitesimal rates of this chain.
(c) Show that this chain is time reversible, and find the limiting probabilities.

Check back soon!

Problem 32

Customers arrive at a two-server station in accordance with a Poisson process having rate $\lambda$. Upon arriving, they join a single queue. Whenever a server completes a service, the person first in line enters service. The service times of server $i$ are exponential with rate $\mu_i, i=1,2$, where $\mu_1+\mu_2>\lambda$. An arrival finding both servers free is equally likely to go to either one. Define an appropriate continuous-time Markov chain for this model, show it is time reversible, and find the limiting probabilities.

Check back soon!

Problem 33

Consider two $M / M / 1$ queues with respective parameters $\lambda_i, \mu_i, i=1,2$. Suppose they share a common waiting room that can hold at most three customers. That is, whenever an arrival finds her server busy and three customers in the waiting room, she goes away. Find the limiting probability that there will be $n$ queue 1 customers and $m$ queue 2 customers in the system.

Hint: Use the results of Exercise 28 together with the concept of truncation.

Check back soon!

Problem 34

Four workers share an office that contains four telephones. At any time, each worker is either "working" or "on the phone." Each "working" period of worker $i$ lasts for an exponentially distributed time with rate $\lambda_i$, and each "on the phone" period lasts for an exponentially distributed time with rate $\mu_i, i=1,2,3,4$.
(a) What proportion of time are all workers "working"?
Let $X_i(t)$ equal 1 if worker $i$ is working at time $t$, and let it be 0 otherwise. Let $\mathbf{X}(t)=\left(X_1(t), X_2(t), X_3(t), X_4(t)\right)$.
(b) Argue that $\{\mathbf{X}(t), t \geqslant 0\}$ is a continuous-time Markov chain and give its infinitesimal rates.
(c) Is $\{\mathbf{X}(t)\}$ time reversible? Why or why not?
Suppose now that one of the phones has broken down. Suppose that a worker who is about to use a phone but finds them all being used begins a new "working" period.
(d) What proportion of time are all workers "working"?

Check back soon!

Problem 35

Consider a time reversible continuous-time Markov chain having infinitesimal transition rates $q_{i j}$ and limiting probabilities $\left\{P_i\right\}$. Let $A$ denote a set of states for this chain, and consider a new continuous-time Markov chain with transition rates $q_{i j}^*$ given by
$$
q_{i j}^*= \begin{cases}c q_{i j}, & \text { if } i \in A, j \notin A \\ q_{i j}, & \text { otherwise }\end{cases}
$$
where $c$ is an arbitrary positive number. Show that this chain remains time reversible, and find its limiting probabilities.

Check back soon!

Problem 36

Consider a system of $n$ components such that the working times of component $i, i=1, \ldots, n$, are exponentially distributed with rate $\lambda_i$. When failed, however, the repair rate of component $i$ depends on how many other components are down. Specifically, suppose that the instantaneous repair rate of component $i, i=1, \ldots, n$, when there are a total of $k$ failed components, is $\alpha^k \mu_i$.
(a) Explain how we can analyze the preceding as a continuous-time Markov chain. Define the states and give the parameters of the chain.
(b) Show that, in steady state, the chain is time reversible and compute the limiting probabilities.

Check back soon!
04:55

Problem 37

For the continuous-time Markov chain of Exercise 3 present a uniformized version.

Alvar Garcia-Fernandez
Alvar Garcia-Fernandez
Numerade Educator

Problem 38

In Example 6.20, we computed $m(t)=E[O(t)]$, the expected occupation time in state 0 by time $t$ for the two-state continuous-time Markov chain starting in state 0 . Another way of obtaining this quantity is by deriving a differential equation for it.
(a) Show that
$$
m(t+h)=m(t)+P_{00}(t) h+o(h)
$$
(b) Show that
$$
m^{\prime}(t)=\frac{\mu}{\lambda+\mu}+\frac{\lambda}{\lambda+\mu} e^{-(\lambda+\mu) t}
$$
(c) Solve for $m(t)$.

Check back soon!

Problem 39

Let $O(t)$ be the occupation time for state 0 in the two-state continuous-time Markov chain. Find $E[O(t) \mid X(0)=1]$.

Check back soon!

Problem 40

Consider the two-state continuous-time Markov chain. Starting in state 0 , find $\operatorname{Cov}[X(s), X(t)]$

Check back soon!

Problem 41

Let $Y$ denote an exponential random variable with rate $\lambda$ that is independent of the continuous-time Markov chain $\{X(t)\}$ and let
$$
\bar{P}_{i j}=P\{X(Y)=j \mid X(0)=i\}
$$
(a) Show that
$$
\bar{P}_{i j}=\frac{1}{v_i+\lambda} \sum_k q_{i k} \bar{P}_{k j}+\frac{\lambda}{v_i+\lambda} \delta_{i j}
$$
where $\delta_{i j}$ is 1 when $i=j$ and 0 when $i \neq j$.
(b) Show that the solution of the preceding set of equations is given by
$$
\overline{\mathbf{P}}=(\mathbf{I}-\mathbf{R} / \lambda)^{-1}
$$
where $\overline{\mathbf{P}}$ is the matrix of elements $\bar{P}_{i j}, \mathbf{I}$ is the identity matrix, and $\mathbf{R}$ the matrix specified in Section 6.8 .
(c) Suppose now that $Y_1, \ldots, Y_n$ are independent exponentials with rate $\lambda$ that are independent of $\{X(t)\}$. Show that
$$
P\left\{X\left(Y_1+\cdots+Y_n\right)=j \mid X(0)=i\right\}
$$
is equal to the element in row $i$, column $j$ of the matrix $\overline{\mathbf{P}}^n$.
(d) Explain the relationship of the preceding to Approximation 2 of Section 6.8 .

Check back soon!
04:48

Problem 42

(a) Show that Approximation 1 of Section 6.8 is equivalent to uniformizing the continuous-time Markov chain with a value $v$ such that $v t=n$ and then approximating $P_{i j}(t)$ by $P_{i j}^{* n}$.
(b) Explain why the preceding should make a good approximation.
Hint: What is the standard deviation of a Poisson random variable with mean $n$ ?

Abhirup Pal
Abhirup Pal
Numerade Educator