Consider a linear mapping y = Wx with y ∈ R^2, x ∈ R^2 and W ∈ R^2x2 with
W = [1 2]
Then the singular values of W are σ1 = 3, σ2 = 1. What is the condition number of W? Why does the condition number matter? (3 marks)
2. Corrected_text: Consider a linear mapping y = Wx with y ∈ R^2, x ∈ R^2 and W ∈ R^2x2 with W = [0.1 0.2; 0.1 0.3]
Perform one gradient descent update step to W with a learning rate of α = 0.1 with the data point (x,yT) = ([1,1]T, [1,2]T) (one data point means we are doing "online learning"). Use the loss function L(W) = ||y - W||^2 (6 marks).
3. Corrected_text: Describe the purpose of using momentum as opposed to vanilla gradient descent. Explain the relevant problem with gradient descent and how momentum fixes this problem. (4 marks)