Consider all 40 observations on the delivery time data....

Consider all 40 observations on the delivery time data. Delete $10 \%$ (4) of the observations at random. Fit a model to the remaining 36 observations, predict the four deleted values, and calculate $R^{2}$ for prediction. Repeat these calculations 100 times. Calculate the average $R^{2}$ for prediction. What information does this convey about the predictive capability of the model? How does the average of the $100 R^{2}$ for prediction values compare to $R^{2}$ for prediction based on PRESS for all 40 observations?

Key Concepts

Hold-out Validation

This concept involves setting aside a portion of data to test a model’s predictions rather than evaluating the model solely on the training data. By deleting a subset of observations and predicting their values, one assesses how well the model generalizes beyond the data it was fitted on, which is crucial for understanding its practical predictive capability.

Predictive R-squared

Predictive R-squared quantifies how well the model’s predictions match actual outcomes in the test set, rather than representing just the fit to the training data. It provides a performance metric for prediction and indicates the proportion of variance in the deleted (or held-out) observations that is explained by the model.

Monte Carlo or Random Cross-Validation

In this context, repeating the process 100 times using randomly selected hold-out sets is a form of Monte Carlo cross-validation. This repeated random sampling provides a robust estimate of the model’s predictive performance by averaging out variability from different random splits, thereby giving insight into the stability and reliability of the predictions.

PRESS (Prediction Sum of Squares)

PRESS is a statistic used to assess a model’s predictive accuracy by calculating the sum of squared differences between the observed and predicted values for each omitted observation. It is often used in leave-one-out cross-validation and provides a method to compute an overall R-squared measure based on prediction errors, serving as a benchmark to compare against the average predictive R-squared from repeated hold-out samples.

Please give Ace some feedback