In k-nearest neighbors (k-NN), the classification is achieved by majority
vote in the vicinity of data. Given n points, imagine two classes of data each of n/2 points,
which are overlapped to some extent in a 2-dimensional space.
(a) Describe what happens to the training error (using all available data) when
the neighbor size k varies from n to 1.
(b)Predict and explain with a sketch how the generalization error (e.g. holding
out some data for testing) would change when k varies? Explain your reasoning.
(c) Is it possible to build a univariate decision tree (with decisions at each
node of the form “is x > a”, “is x < b”, “is y > c”, or “is y < d” for any real constants
a, b, c, d) which classifies exactly similar to a 1-NN using the Euclidean distance mea-
sure? If so, explain how. If not, explain why not.