A Single Test Split Isn't Enough to Compare Models

Most people learn that a single train/test split can make a model look better than it is. That's half the problem. The half that gets missed: the same split, in the same experiment, can simultaneously make a competing model look worse than it is. You end up with two wrong numbers pointing in opposite directions, and the gap between them looks like a decisive result.

Here is what that looks like with real numbers, and how to catch it.

The setup

Classify handwritten 5s and 6s. 1,220 images, 256 grayscale values each. Nine candidate models: one linear regression, and KNN at k = 1, 3, 5, 7, 9, 11, 13, 15.

Start with training error, and understand why it can't answer your question

library(class)
kks <- c(1, 3, 5, 7, 9, 11, 13, 15)
for (kk in kks) {
  ypred <- knn(ziptrain56[,-1], ziptrain56[,-1], ziptrain56[,1], k=kk)
  print(mean(ypred != ziptrain56[,1]))
}

Linear regression lands at 0.49%. KNN at k=1 returns exactly zero.

Do not read that zero as performance. With one neighbor, the nearest point to any training image is itself, so k=1 is structurally incapable of being wrong on data it has already seen. You will get 0% on every dataset you ever run it against. When a metric is guaranteed by the algorithm's mechanics, it carries no information about the data.

This is the general form of the problem: training error tells you how well a model memorized. You want to know how well it predicts. Those are different questions, and only one of them matters.

Add a held-out test set, and watch it produce a confident lie

ypred <- knn(ziptrain56[,-1], ziptest56[,-1], ziptrain56[,1], k=3)
mean(ypred != ziptest56[,1])

A 16x difference. Most people stop here, and the conclusion writes itself.

Before you accept it, look at KNN k=3 across both tables. It scored 0.30% on data it had never seen and 0.49% on data it had memorized. A model does not get sharper on unfamiliar data. When testing error comes in below training error, you are not looking at a strong model. You are looking at a test set that happened to draw an easy hand.

That is your signal to stop trusting the number and start repeating the experiment.

Repeat the split 100 times

Monte Carlo cross validation: reshuffle which rows are training and which are testing, refit every model, record the errors, repeat.

zip56full <- rbind(ziptrain56, ziptest56)
n1 <- dim(ziptrain56)[1]; n <- dim(zip56full)[1]
set.seed(7406)

TEALL <- NULL
for (b in 1:100) {
  flag <- sort(sample(1:n, n1))
  train_b <- zip56full[flag, ]; test_b <- zip56full[-flag, ]

  regmod <- lm(V1 ~ ., data = train_b)
  te_lr <- mean((5 + (predict.lm(regmod, test_b[,-1]) >= 5.5)) != test_b[,1])

  te_knn <- sapply(kks, function(kk) {
    mean(knn(train_b[,-1], test_b[,-1], train_b[,1], k=kk) != test_b[,1])
  })
  TEALL <- rbind(TEALL, c(te_lr, te_knn))
}
apply(TEALL, 2, mean); apply(TEALL, 2, var)

Both estimates moved, in opposite directions. KNN k=3 nearly tripled, from 0.30% to 0.87%. Linear regression fell by half, from 4.85% to 2.53%. One split was generous to the first model and punishing to the second, which is exactly how a 3x difference gets reported as 16x.

Note that the ranking survived. KNN k=3 is genuinely the better model here. Repeating the experiment did not overturn the conclusion, it corrected the magnitude. That distinction matters when you are deciding how much to invest in a model choice.

Now read the variance column, which most people skip. KNN k=3 has the tightest spread of the nine at 1.89e-05, meaning its error barely moves regardless of how the data is partitioned. That is a separate finding from having the lowest average, and in production it is frequently the more valuable one. Consistency is a feature. A model that is occasionally excellent and occasionally poor is harder to build on than one that is reliably good.

The rules worth carrying forward

Training error and testing error answer different questions. Across the 100 runs, linear regression's training error was 0.53%, better than KNN at k=5 and above. Its testing error was the worst of all nine models. Both figures are correct. Only one of them tells you what happens next.

A guaranteed metric is not a discovery. k=1 hitting 0% training error is a property of the algorithm, not evidence about the data. Learn to recognize which of your metrics are structurally determined before you interpret them.

Scope your conclusions to the evidence. KNN k=3 won on this data, against these nine methods. "KNN beats linear regression" is a substantially larger claim than the experiment supports.

Treat weak results as diagnostic. Linear regression did not fail arbitrarily here. Adjacent pixels in a handwritten digit have similar values, so the 256 features are heavily correlated, and linear regression assumes they are not. That is multicollinearity, and it points to a specific remedy: feature selection or dimensionality reduction. Dismissing a poor result as "that model doesn't work" discards the most useful information the experiment produced.

When a result looks unusually clean, run it again. A flattering number is a hypothesis. Treat it as one until the average stops moving.