(Regression) A common use of optimization is for the purpose of function approximation. Suppose, for example, that through an experiment the value of a function $g$ is observed at $m$ points, $x_1, x_2, \dots, x_m$. Thus, values $g(x_1), g(x_2), \dots, g(x_m)$ are known. We wish to approximate the function by a polynomial
$h(x) = a_nx^n + a_{n-1}x^{n-1} + \dots + a_0$
of degree $n$ (or less), where $n < m$. Corresponding to any choice of the approximating polynomial, there will be a set of errors $e_k = g(x_k) - h(x_k)$. We define the best approximation as the polynomial that minimizes the sum of the squares of these errors; that is, minimizes
$\sum_{k=1}^{m} (e_k)^2$.
This in turn means that we minimize
$f(a) = \sum_{k=1}^{m} [g(x_k) - (a_nx^n + a_{n-1}x^{n-1} + \dots + a_0)]^2$
with respect to $a = (a_0, a_1, \dots, a_n)$ to find the best coefficients.
(a) Let $q_{ij} = \sum_{k=1}^{m} (x_k)^{i+j}$, $b_j = \sum_{k=1}^{m} g(x_k)(x_k)^j$ and $c = \sum_{k=1}^{m} g(x_k)^2$. Show that $f(a)$ can be written in a compact form $f(a) = a^TQa - 2b^Ta + c$ where $Q = [q_{ij}]$, $b = (b_1, b_2, \dots, b_{n+1})$.
(b) What are the first-order necessary conditions?
(c) Suppose we want to use a second order polynomial to fit some function $g(x) = [1, 1.22, 1.49, 2.72, 7.39]$, at the points $x = [0, 1, 2, 5, 10]$. Find the coefficients $a$.
