Question

Problem 4. (50 points) Consider a multivariate linear regression problem of mapping $mathbb{R}^d$ to $mathbb{R}$, with two different objective functions. The first objective function is the sum of squared errors, as presented in class; i.e., $sum_{i=1}^n e_i^2$, where $e_i = w_0 + sum_{j=1}^d w_j x_{ij} - y_i$. The second objective function is the sum of square Euclidean distances to the hyperplane; i.e., $sum_{i=1}^n r_i^2$, where $r_i$ is the Euclidean distance between point $(x_i, y_i)$ to the hyperplane $f(x) = w_0 + sum_{j=1}^d w_j x_j$. a) (10 points) Derive a gradient descent algorithm to find the parameters of the model that minimizes the sum of squared errors. b) (20 points) Derive a gradient descent algorithm to find the parameters of the model that minimizes the sum of squared distances. c) (20 points) Implement both algorithms and test them on 3 different datasets. Datasets can be randomly generated, as in class, or obtained from resources such as UCI Machine Learning Repository. Compare the solutions to the closed-form (maximum likelihood) solution derived in class and find $R^2$ in all cases on the same dataset used to fit the parameters; i.e., do not implement cross-validation. Briefly describe the data you use and discuss your results.

          Problem 4. (50 points) Consider a multivariate linear regression problem of mapping $mathbb{R}^d$ to $mathbb{R}$, with two different objective functions. The first objective function is the sum of squared errors, as presented in class; i.e., $sum_{i=1}^n e_i^2$, where $e_i = w_0 + sum_{j=1}^d w_j x_{ij} - y_i$. The second objective function is the sum of square Euclidean distances to the hyperplane; i.e., $sum_{i=1}^n r_i^2$, where $r_i$ is the Euclidean distance between point $(x_i, y_i)$ to the hyperplane $f(x) = w_0 + sum_{j=1}^d w_j x_j$.
a) (10 points) Derive a gradient descent algorithm to find the parameters of the model that minimizes the sum of squared errors.
b) (20 points) Derive a gradient descent algorithm to find the parameters of the model that minimizes the sum of squared distances.
c) (20 points) Implement both algorithms and test them on 3 different datasets. Datasets can be randomly generated, as in class, or obtained from resources such as UCI Machine Learning Repository. Compare the solutions to the closed-form (maximum likelihood) solution derived in class and find $R^2$ in all cases on the same dataset used to fit the parameters; i.e., do not implement cross-validation. Briefly describe the data you use and discuss your results.

Problem 4. (50 points) Consider a multivariate linear regression problem of mapping mathbbR^d to mathbbR, with two different objective functions. The first objective function is the sum of squared errors, as presented in class; i.e., sumi=1^n ei^2, where ei = w0 + sumj=1^d wj xij - yi. The second objective function is the sum of square Euclidean distances to the hyperplane; i.e., sumi=1^n ri^2, where ri is the Euclidean distance between point (xi, yi) to the hyperplane f(x) = w0 + sumj=1^d wj xj.
a) (10 points) Derive a gradient descent algorithm to find the parameters of the model that minimizes the sum of squared errors.
b) (20 points) Derive a gradient descent algorithm to find the parameters of the model that minimizes the sum of squared distances.
c) (20 points) Implement both algorithms and test them on 3 different datasets. Datasets can be randomly generated, as in class, or obtained from resources such as UCI Machine Learning Repository. Compare the solutions to the closed-form (maximum likelihood) solution derived in class and find R^2 in all cases on the same dataset used to fit the parameters; i.e., do not implement cross-validation. Briefly describe the data you use and discuss your results.

Added by Lisa T.

Question

Please give Ace some feedback