3. Given a vocabulary of size |V|, we model the probability of the next word $x_{t+1}$ given the previous sequence $x_1, \dots, x_t$ as:
$$P(x_{t+1} = v_j | x_1, \dots, x_t)$$
The RNN computes the following at each time step $t$:
$$e^{(t)} = x^{(t)} L \qquad (1)$$
$$h^{(t)} = \text{sigmoid}(h^{(t-1)} H + e^{(t)} I + b_1) \qquad (2)$$
$$\hat{y}^{(t)} = \text{softmax}(h^{(t)} U + b_2) \qquad (3)$$
Where:
* $L \in \mathbb{R}^{|V| \times d}$ is the word embedding matrix
* $H, I \in \mathbb{R}^{D_h \times D_h}$ are weight matrices
* $U \in \mathbb{R}^{D_h \times |V|}$ is the output projection matrix
* $b_1 \in \mathbb{R}^{D_h}, b_2 \in \mathbb{R}^{|V|}$ are biases
Cross-entropy loss:
$$J^{(t)} = - \sum_{i=1}^{|V|} y_i^{(t)} \log \hat{y}_i^{(t)}$$
(a) Show that perplexity is the exponential of the cross-entropy loss. Compute CE and PP for $|V| = 2000$ and $|V| = 10000$ if predictions are uniform.
(b) Compute the gradients of the loss at time $t$ with respect to:
* $\frac{\partial J^{(t)}}{\partial U}$ and $\frac{\partial J^{(t)}}{\partial b_2}$
* $\frac{\partial J^{(t)}}{\partial h^{(t)}$ and $\frac{\partial J^{(t)}}{\partial I}$
* $\frac{\partial J^{(t)}}{\partial H}$, $\frac{\partial J^{(t)}}{\partial b_1}$ and $\frac{\partial J^{(t)}}{\partial L^{(t)}}$
(c) Unroll the RNN for 3 time steps and write the backpropagation-through-time equations for each parameter.
(d) Compute the time complexity for one forward and backward step. What is the slowest operation?