Consider a neural network shown in the figure below. We have three inputs and two outputs.
The activation function used in hidden layer is the ReLU function and no activation function
is used at the output layer. The loss function chosen is given by: $J = \frac{1}{2}||y - \hat{y}||_2^2$, where $y$
is the true training label vector, and $\hat{y}$ is the predicted output vector. You are given the
following transformations:
(a) [3 marks] Define the weight matrices and bias vectors for each layer using the ap-
propriate notation and specify their sizes.
(b) [5 marks] Formulate a set of equations for Forward pass.
(c) [2 marks] Calculate the total number of trainable parameters of the neural network.
(d) [5 marks] Derive $\frac{\partial J}{\partial w_{ij}^{l}}$
(e) [5 marks] Now write $\frac{\partial J}{\partial W}$, i.e., use vector products.
(f) [10 marks] Using your result from previous parts, derive $\frac{\partial J}{\partial w_{ij}^{l}}$
