Question

Problem 1: Computational Graphs and Backpropagation (25 marks) Given a sequential dataset D = {(x1, y1), ..., (xT, yT)}, consider a vanilla RNN model, which computes the loss function at the t-th time step using the following formulas: a_t = b + Wh_{t-1} + Ux_t, h_t = tanh(a_t), o_t = c + v h_t, L_t = (o_t - y_t)^2, where $tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$ and the final loss function is given by TrainLoss = $\sum_{t=1}^T L_t$ with T being the length of the sequence. For simplicity, let's assume the input data is three dimensional (i.e., x ? R³), the hidden size is 2 (i.e., h ? R²) and T = 1 in this problem. Q1. Construct the computational graph for L1. Q2. What are the model parameters to be learned? Q3. Given the following model parameters and data, compute the loss function in forward pass and compute the gradient with respect to all model parameters in backward pass. We follow the common practice to use the zero initial hidden states, i.e., h0 = [0, 0]^T. b = [0.1, 0]^T W = \begin{bmatrix} 0.5 & -0.4 \ -0.3 & 0.7 \ \end{bmatrix} U = \begin{bmatrix} 0.8 & -0.3 & 0.5 \ -1 & 0.1 & 0.9 \ \end{bmatrix} c = 0.7 v = [0.5, 0.8]^T D = {([1, -1, 1], 1.5)}

          Problem 1: Computational Graphs and Backpropagation (25 marks)
Given a sequential dataset D = {(x1, y1), ..., (xT, yT)}, consider a vanilla RNN model, which computes the loss function at the t-th time step using the following formulas:
a_t = b + Wh_{t-1} + Ux_t,
h_t = tanh(a_t),
o_t = c + v h_t,
L_t = (o_t - y_t)^2,
where $tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$ and the final loss function is given by TrainLoss = $\sum_{t=1}^T L_t$ with T being the length of the sequence.
For simplicity, let's assume the input data is three dimensional (i.e., x ? R³), the hidden size is 2 (i.e., h ? R²) and T = 1 in this problem.
Q1. Construct the computational graph for L1.
Q2. What are the model parameters to be learned?
Q3. Given the following model parameters and data, compute the loss function in forward pass and compute the gradient with respect to all model parameters in backward pass. We follow the common practice to use the zero initial hidden states, i.e., h0 = [0, 0]^T.
b = [0.1, 0]^T
W = \begin{bmatrix} 0.5 & -0.4 \ -0.3 & 0.7 \ \end{bmatrix}
U = \begin{bmatrix} 0.8 & -0.3 & 0.5 \ -1 & 0.1 & 0.9 \ \end{bmatrix}
c = 0.7
v = [0.5, 0.8]^T
D = {([1, -1, 1], 1.5)}

Problem 1: Computational Graphs and Backpropagation (25 marks)
Given a sequential dataset D = (x1, y1), ..., (xT, yT), consider a vanilla RNN model, which computes the loss function at the t-th time step using the following formulas:
at = b + Wht-1 + Uxt,
ht = tanh(at),
ot = c + v ht,
Lt = (ot - yt)^2,
where tanh(z) = (e^z - e^-z)/(e^z + e^-z) and the final loss function is given by TrainLoss = ∑t=1^T Lt with T being the length of the sequence.
For simplicity, let's assume the input data is three dimensional (i.e., x ? R³), the hidden size is 2 (i.e., h ? R²) and T = 1 in this problem.
Q1. Construct the computational graph for L1.
Q2. What are the model parameters to be learned?
Q3. Given the following model parameters and data, compute the loss function in forward pass and compute the gradient with respect to all model parameters in backward pass. We follow the common practice to use the zero initial hidden states, i.e., h0 = [0, 0]^T.
b = [0.1, 0]^T
W =
< b m a t r i x >

U =
< b m a t r i x >

c = 0.7
v = [0.5, 0.8]^T
D = ([1, -1, 1], 1.5)

Added by Domingo S.

Question

Please give Ace some feedback