Question

8-You are asked to evaluate the performance of two classification models, M1 and M2. The test set you have chosen contains 26 binary attributes, labeled as A through Z. The following table shows the posterior probabilities obtained by applying the models to the test set. (Only the posterior probabilities for the positive class are shown). | Instance | True Class | P(+|A,..., Z, M1) | P(+|A,..., Z, M2) | |---|---|---|---| | 1 | + | 0.73 | 0.61 | | 2 | + | 0.69 | 0.03 | | 3 | - | 0.44 | 0.68 | | 4 | - | 0.55 | 0.31 | | 5 | + | 0.67 | 0.45 | | 6 | + | 0.47 | 0.09 | | 7 | - | 0.08 | 0.38 | | 8 | - | 0.15 | 0.05 | | 9 | + | 0.45 | 0.01 | | 10 | - | 0.35 | 0.04 | As this is a two-class problem, P(-) = 1 - P(+) P(-|A,..., Z) = 1 - P(+|A,..., Z). Assume that we are mostly interested in detecting instances from the positive class. A. Plot the ROC curve for both M1 and M2. (You should plot them on the same graph.) Which model do you think is better? Explain your reasons. B. For model M1, suppose you choose the cutoff threshold to be t = 0.5. In other words, any test instances whose posterior probability is greater than t will be classified as a positive example. Compute the precision, recall, and F-measure for the model at this threshold value.

          8-You are asked to evaluate the performance of two classification models, M1 and M2. The test set you have chosen contains 26 binary attributes, labeled as A through Z.

The following table shows the posterior probabilities obtained by applying the models to the test set. (Only the posterior probabilities for the positive class are shown).

| Instance | True Class | P(+|A,..., Z, M1) | P(+|A,..., Z, M2) |
|---|---|---|---|
| 1 | + | 0.73 | 0.61 |
| 2 | + | 0.69 | 0.03 |
| 3 | - | 0.44 | 0.68 |
| 4 | - | 0.55 | 0.31 |
| 5 | + | 0.67 | 0.45 |
| 6 | + | 0.47 | 0.09 |
| 7 | - | 0.08 | 0.38 |
| 8 | - | 0.15 | 0.05 |
| 9 | + | 0.45 | 0.01 |
| 10 | - | 0.35 | 0.04 |

As this is a two-class problem,
P(-) = 1 - P(+)    P(-|A,..., Z) = 1 - P(+|A,..., Z).

Assume that we are mostly interested in detecting instances from the positive class.

A. Plot the ROC curve for both M1 and M2. (You should plot them on the same graph.) Which model do you think is better? Explain your reasons.
B. For model M1, suppose you choose the cutoff threshold to be t = 0.5. In other words, any test instances whose posterior probability is greater than t will be classified as a positive example. Compute the precision, recall, and F-measure for the model at this threshold value.

8-You are asked to evaluate the performance of two classification models, M1 and M2. The test set you have chosen contains 26 binary attributes, labeled as A through Z.

The following table shows the posterior probabilities obtained by applying the models to the test set. (Only the posterior probabilities for the positive class are shown).

| Instance | True Class | P(+|A,..., Z, M1) | P(+|A,..., Z, M2) |
|—|—|—|—|
| 1 | + | 0.73 | 0.61 |
| 2 | + | 0.69 | 0.03 |
| 3 | - | 0.44 | 0.68 |
| 4 | - | 0.55 | 0.31 |
| 5 | + | 0.67 | 0.45 |
| 6 | + | 0.47 | 0.09 |
| 7 | - | 0.08 | 0.38 |
| 8 | - | 0.15 | 0.05 |
| 9 | + | 0.45 | 0.01 |
| 10 | - | 0.35 | 0.04 |

As this is a two-class problem,
P(-) = 1 - P(+) P(-|A,..., Z) = 1 - P(+|A,..., Z).

Assume that we are mostly interested in detecting instances from the positive class.

A. Plot the ROC curve for both M1 and M2. (You should plot them on the same graph.) Which model do you think is better? Explain your reasons.
B. For model M1, suppose you choose the cutoff threshold to be t = 0.5. In other words, any test instances whose posterior probability is greater than t will be classified as a positive example. Compute the precision, recall, and F-measure for the model at this threshold value.

Added by Matthew S.

Question

Please give Ace some feedback