I'm sorry, I cannot fulfill that request.
Added by Emilia H.
Close
Step 1
Is it due to limitations, regulations, or other factors? Show more…
Show all steps
Your feedback will help us improve your experience
Patha Sharma and 101 other Calculus 3 educators are ready to help you.
Ask a new question
Labs
Want to see this concept in action?
Explore this concept interactively to see how it behaves as you change inputs.
Key Concepts
Recommended Videos
4.57 Capacity of a communication channel. We consider a communication channel, with input X(t) ∈ {1, . . . , n}, and output Y(t) ∈ {1, . . . , m}, for t = 1, 2, . . . (in seconds, say). The relation between the input and the output is given statistically: pij = prob(Y(t) = i|X(t) = j), i = 1, . . . , m, j = 1, . . . , n. The matrix P ∈ R^m!n is called the channel transition matrix, and the channel is called a discrete memoryless channel. A famous result of Shannon states that information can be sent over the communication channel, with arbitrarily small probability of error, at any rate less than a number C, called the channel capacity, in bits per second. Shannon also showed that the capacity of a discrete memoryless channel can be found by solving an optimization problem. Assume that X has a probability distribution denoted x ∈ R^n, i.e., xj = prob(X = j), j = 1, . . . , n. The mutual information between X and Y is given by I(X; Y) = ∑_{i=1}^m ∑_{j=1}^n xj pij log2(pij / ∑_{k=1}^n xk pik). Then the channel capacity C is given by C = sup_x I(X; Y), where the supremum is over all possible probability distributions for the input X, i.e., over x ≵ 0, 1^T x = 1. Show how the channel capacity can be computed using convex optimization. Hint. Introduce the variable y = Px, which gives the probability distribution of the output Y, and show that the mutual information can be expressed as I(X; Y) = c^T x − ∑_{i=1}^m yi log2 yi, where cj = ∑_{i=1}^m pij log2 pij, j = 1, . . . , n. Solution. The capacity is the optimal value of the problem maximize f0(x) = ∑_{i=1}^m ∑_{j=1}^n xj pij log(pij / ∑_{k=1}^n xk pik) subject to x ≵ 0, 1^T x = 1, with variable x. It is possible to argue directly that the objective f0 (which is the mutual information between X and Y) is concave in x. This can be done several ways, starting from the example 3.19. Another (related) approach is to follow the hint given, and introduce y = Px as another variable. We can express the mutual information in terms of x and y as I(X; Y) = ∑_{i,j} xj pij log(pij / ∑_k xk pik) = ∑_j xj ∑_i pij log pij − ∑_i yi log yi = −c^T x − ∑_i yi log yi, where cj = −∑_i pij log pij. Therefore the channel capacity problem can be expressed as maximize I(X; Y) = −c^T x − ∑_i yi log yi subject to x ≵ 0, 1^T x = 1 y = Px, with variables x and y. The objective is a constant plus the entropy of y, hence concave, so this is a convex optimization problem.
Patha S.
Period Doubling and Chaos In the study of dynamical systems, the phenomena of period doubling and chaos are observed. These phenomena can be seen when one uses a numerical scheme to approximate the solution to an initial value problem for a nonlinear differential equation such as the following logistic model for population growth: (13) dp/dt = 10p(1 - p), p(0) = 0.1 . (See Section 3.2.) (a) Solve the initial value problem (13) and show that p(t) approaches 1 as t -> +inf. (b) Show that using Euler's method (see Sections 1.4 and 3.6) with step size h to approximate the solution to (13) gives (14) pn+1 = (1 + 10h)pn - (10h)pn^2, p0 = 0.1 . (c) For h = 0.18, 0.23, 0.25, and 0.3, show that the first 40 iterations of (14) appear to (i) converge to 1 when h = 0.18, (ii) jump between 1.18 and 0.69 when h = 0.23, (iii) jump between 1.23, 0.54, 1.16, and 0.70 when h = 0.25, and (iv) display no discernible pattern when h = 0.3. The transitions from convergence to jumping between two numbers, then four numbers, and so on, are called period doubling. The phenomenon displayed when h = 0.3 is referred to as chaos. This transition from period doubling to chaos as h increases is frequently observed in dynamical systems. The transition to chaos is nicely illustrated in the bifurcation diagram (see Figure 3.20). This diagram is generated for equation (14) as follows. Beginning at h = 0.18, compute the sequence {pn} using (14) and, starting at n = 201, plot the next 30 values-that is, p201, p202, ..., p230. Next, increment h by 0.001 to 0.181 and repeat. Continue this process until h = 0.30. Notice how the figure splits from one branch to two, then four, and then finally gives way to chaos. Our concern is with the instabilities of the numerical procedure when h is not chosen small enough. Fortunately, the instability observed for Euler's method-the period doubling and chaos-was immediately recognized because we know that this type of behavior is not expected of a solution to the logistic equation. Consequently, if we had tried Euler's method with h = 0.23, 0.25, or 0.3 to solve (13) numerically, we would have realized that h was not chosen small enough. The situation for the classical fourth-order Runge-Kutta method (see Section 3.7) is more troublesome. It may happen that for a certain choice of h period doubling occurs, but it is also possible that for other choices of h the numerical solution actually converges to a limiting value that is not the limiting value for any solution to the logistic equation in (13). (d) Approximate the solution to (13) by computing the first 60 iterations of the classical fourth-order Runge-Kutta method using the step size h = 0.3. (Thus, for the subroutine on page 135, input N = 60 and c = 60(0.3) = 18.) Repeat with h = 0.325 and h = 0.35. Which values of h (if any) do you feel are giving the "correct" approximation to the solution? Why?
Sri K.
Problem 3 - Coding question [30 points] Safer path Optimal path The Cliff R = -1 R = -100 Figure 1: Cliff-walking or gridworld problem (Example 6.6 in Sutton and Barto's book) Consider the gridworld shown in Fig. 1. This is a standard undiscounted (γ = 1), episodic task, with start (S) and goal (G) states, and the usual actions causing movement up, down, right, and left. Reward is -1 on all transitions except those into the region marked "The Cliff." Stepping into this region incurs a reward of -100 and sends the agent instantly back to the start. There are 48 states (positions in the grid) and 4 actions. Let μ be a fixed stochastic policy, which assigns uniform distribution on ᶜ, i.e., given any state i, we have the probability of taking action a is μ(a|i) = 1/4 for all a ∈ ᶜ. Your job is to implement TD(λ) for finding Vᵡ. You can either implement TD(λ) with tabular settings or with linear function approximations [1]. Note that this is an undiscounted and episodic problem. [1.] Value Function Approximation in Reinforcement Learning using the Fourier Basis. George Konidaris and Sarah Osentoski and Philip Thomas. Questions: Write a simulation program to compute Vᵡ for the cliff-walking problem using TD(λ). In your simulation, consider 1000 episodes, where each episode runs 20 steps of TD(λ) given in class. For each episode, compute the norm of the expected TD update (NEU), the average of temporal difference, i.e., NEU = 1/20 ∑ (dₖzₖ)², where zₖ is the trace vector. Then for every 10 episode, you take the average of the NEU values and plot this average as a function of the number of episodes. Note that for each episode, you should initialize your function values Vᵡ as the values returned by the previous step. You are asked to submit a pseudo code to explain your simulation and a plot which shows 5 curves of the average of NEU values as a function of the number of episodes for λ = 0, .3, .5, .7, 1. Finally, briefly explain the impacts of λ on the performance of TD learning.
Akash M.
Recommended Textbooks
Calculus: Early Transcendentals
Thomas Calculus
Transcript
Watch the video solution with this free unlock.
EMAIL
PASSWORD