Question

Problem 1: Linear Regression and Gradient Learning [30 points] In class we derived linear regression and various learning algorithms based on gradient descent. In addition to the least square objective, we also learned its probabilistic perspective where each observation is assumed to have a Gaussian noise. (i.e., Noise of each example is an independent and identically distributed sample from a normal distribution) In this problem, you are supposed to deal with the following regression model that includes two linear features and one quadratic feature. y = ??0 + ??1x1 + ??2x2 + ??3x1^2 + ? where ? ~ N(0, ?2) Your goal is to develop a gradient descent learning algorithm that will estimate the best parameters ?? = {??0, ??1, ??2, ??3}. (a) Given the definition of noise, derive the corresponding mean and variance parameters of the normal distribution for y|x1, x2; ??. Write also down its probability density function. (b) You are provided with a training observations D = {(x1(i), x2(i), y(i))|1 ? i ? m}. Derive the conditional log-likelihood that will be later maximized to make D most likley. (c) If you omit all the constant that does not relate to our parameters ??, what will be the objective function J(??0, ??1, ??2, ??3) that you are going to perform Maximum Likelihood Estimation? Does J look similar to the Least Square objective for this problem? (d) Compute the gradient of J(??) with respect to each parameter. (Hint: You should evaluate the partial derivatives of J(??) with respect to each ??j for 0 ? j ? 3) (e) [Coding] Develop two learning algorithms from the scratch: batch and stochastic gradient descent for this problem on the Auto dataset given in the Problem 4 in Homework 1. Compare and contrast the performance among your batch gradient, stochastic gradient, and R's built-in function call: lm. Are the two best input features for predicting the output mpg same across different algorithms? (Hint: At least your stochastic gradient algorithm must learn parameters ?? comparable to the result from calling the R's built-in function. Otherwise try to tune the learning rate ? < 1.0)

          Problem 1: Linear Regression and Gradient Learning [30 points]
In class we derived linear regression and various learning algorithms based on gradient descent. In addition to the least square objective, we also learned its probabilistic perspective where each observation is assumed to have a Gaussian noise. (i.e., Noise of each example is an independent and identically distributed sample from a normal distribution) In this problem, you are supposed to deal with the following regression model that includes two linear features and one quadratic feature.
y = ??0 + ??1x1 + ??2x2 + ??3x1^2 + ? where ? ~ N(0, ?2)
Your goal is to develop a gradient descent learning algorithm that will estimate the best parameters ?? = {??0, ??1, ??2, ??3}.
(a) Given the definition of noise, derive the corresponding mean and variance parameters of the normal distribution for y|x1, x2; ??. Write also down its probability density function.
(b) You are provided with a training observations D = {(x1(i), x2(i), y(i))|1 ? i ? m}. Derive the conditional log-likelihood that will be later maximized to make D most likley.
(c) If you omit all the constant that does not relate to our parameters ??, what will be the objective function J(??0, ??1, ??2, ??3) that you are going to perform Maximum Likelihood Estimation? Does J look similar to the Least Square objective for this problem?
(d) Compute the gradient of J(??) with respect to each parameter. (Hint: You should evaluate the partial derivatives of J(??) with respect to each ??j for 0 ? j ? 3)
(e) [Coding] Develop two learning algorithms from the scratch: batch and stochastic gradient descent for this problem on the Auto dataset given in the Problem 4 in Homework 1. Compare and contrast the performance among your batch gradient, stochastic gradient, and R's built-in function call: lm. Are the two best input features for predicting the output mpg same across different algorithms? (Hint: At least your stochastic gradient algorithm must learn parameters ?? comparable to the result from calling the R's built-in function. Otherwise try to tune the learning rate ? < 1.0)

Problem 1: Linear Regression and Gradient Learning [30 points]
In class we derived linear regression and various learning algorithms based on gradient descent. In addition to the least square objective, we also learned its probabilistic perspective where each observation is assumed to have a Gaussian noise. (i.e., Noise of each example is an independent and identically distributed sample from a normal distribution) In this problem, you are supposed to deal with the following regression model that includes two linear features and one quadratic feature.
y = ??0 + ??1x1 + ??2x2 + ??3x1^2 + ? where ? N(0, ?2)
Your goal is to develop a gradient descent learning algorithm that will estimate the best parameters ?? = ??0, ??1, ??2, ??3.
(a) Given the definition of noise, derive the corresponding mean and variance parameters of the normal distribution for y|x1, x2; ??. Write also down its probability density function.
(b) You are provided with a training observations D = (x1(i), x2(i), y(i))|1 ? i ? m. Derive the conditional log-likelihood that will be later maximized to make D most likley.
(c) If you omit all the constant that does not relate to our parameters ??, what will be the objective function J(??0, ??1, ??2, ??3) that you are going to perform Maximum Likelihood Estimation? Does J look similar to the Least Square objective for this problem?
(d) Compute the gradient of J(??) with respect to each parameter. (Hint: You should evaluate the partial derivatives of J(??) with respect to each ??j for 0 ? j ? 3)
(e) [Coding] Develop two learning algorithms from the scratch: batch and stochastic gradient descent for this problem on the Auto dataset given in the Problem 4 in Homework 1. Compare and contrast the performance among your batch gradient, stochastic gradient, and R's built-in function call: lm. Are the two best input features for predicting the output mpg same across different algorithms? (Hint: At least your stochastic gradient algorithm must learn parameters ?? comparable to the result from calling the R's built-in function. Otherwise try to tune the learning rate ? < 1.0)

Added by Matthew D.

Question

Please give Ace some feedback