Let f:R^n->R be a C^2-function and T:R^n->R^n, T(y):=By+b an affine-linear transformation of variables with an invertible matrix B in R^(n x n) and a vector b in R^n. Denote by hat(f)(y):=f(T(y)) the function f transformed to y coordinates. Further let x in R^n be a point with gradf(x)!=0, and x_g denote the result of a gradient step for the solution of the problem min_z in R^n f(z) starting at the point x in R^n, i.e.
x_g:=x-alpha gradf(x),
for some step size alpha >0. Analogously let y_g be the result of a gradient step for hat(f) starting at y=T^(-1)(x), i.e.
y_g:=y-alpha gradhat(f)(y).
The Rosenbrock function is a non-convex function, introduced by Howard H. Rosenbrock (1920-2010) in 1960, which is used as a performance test problem for optimization algorithms.
a) Express gradhat(f)(y) and H_hat(f)(y) in terms of gradf(x), H_f(x), B, and b. Write down your derivation.
b) Show that a step y_g=y-alpha gradhat(f)(y) in the transformed space can be seen as a step in the original space with a search direction s=T(y_g)-x which solves a linear system Ms=-gradf(x). Here, M is a symmetric positive definite matrix.
Hint: Methods of this type are called Quasi-Newton methods and will be discussed later in the lecture.
c) For which class of matrices B is the Gradient Descent Method invariant under the transformation T, i.e. when does T(y_g)=x_g hold (for the same step size alpha)?
d) Now consider specifically quadratic functions of the type
f(x)=c^T x+(1/2)x^T Cx,
with C in R^(n x n) symmetric positive definite and c in R^n. In the lecture it was shown (or will be shown) that for the gradient descent method with the minimization rule for the step size, in order to minimize f of the form (1) we can expect the following rate of convergence:
||x^(k)-ar{x}||<=sqrt((lambda_max(C))/(lambda_min(C)))((lambda_max(C)-lambda_min(C))/(lambda_max(C)+lambda_min(C)))^k||x^(0)-ar{x}||.
Argue, why it is useful to choose B in the transformation T in such a way that the condition number of B^T C B becomes as small as possible, such that the method converges quickly in the y-space. Why is a choice of B with B B^T ≈ C^(-1) especially useful?
Hint: the conditioning number kappa(A) of a symmetric positive definite (s.p.d.) matrix A satisfies: kappa(A)=(lambda_max(A))/(lambda_min(A)).
