Section 1

1 Introduction

A model is built from patients we already have. It is then used on patients we have not seen yet: the next person in clinic, or patients in another hospital. So the real test of a model is not how well it fits the patients it learned from. It is how well it predicts for new patients. A model that does this well is said to generalise.

Twelve patients on a plot of drug dose against fall in blood pressure. A very flexible curve passes close to all of them, with an error of 0.7. Twelve new patients appear in orange, and the curve misses many of them: the error on them is 24.9. A simpler, gently bending curve has an error of 5.2 on the first twelve patients and 7.0 on the new ones.

In the animation, twelve patients took different doses of a blood-pressure drug. A very flexible curve passes close to all twelve. Its error on them (the MSE from lesson 6.1, in mmHg²) is only 0.7. But on twelve new patients its error is 24.9. A simpler curve is a little worse on the first twelve (5.2) and much better on the new ones (7.0).

The flexible curve has memorised the twelve patients, including random ups and downs that will not happen again. A student who memorises the answers to last year's exam can still fail this year's exam. The same is true for models.

What you will be able to do

  • Recognise underfitting and overfitting.
  • Explain why we keep test patients aside, and read a plot of training and test error.
  • Describe bias and variance, and how a penalty on the weights (ridge or lasso) controls overfitting.
Section 2

2 Underfitting and overfitting

The twelve patients follow a typical dose–response pattern: the fall in blood pressure rises quickly at low doses and then levels off. Each patient also has some noise: random variation from the measurement, the time of day, and many things we do not record.

In lesson 1.2 you fitted curves with polynomials. The degree of the polynomial sets how flexible the model is. A degree-1 line cannot bend. A degree-9 polynomial can bend up to eight times.

Three panels with the same twelve patients. A straight line misses the bend in the data, with training error 13.4. A degree-2 curve follows the pattern, with training error 5.2. A degree-9 curve wiggles to reach every patient, with training error 0.7.
  • Underfitting: the model is too simple to follow the real pattern. The straight line misses the bend, so it is wrong in the same way for many patients.
  • Overfitting: the model is so flexible that it follows the noise as well as the pattern. The degree-9 curve wiggles to reach single patients. The wiggles are not real biology, so they will be wrong for the next patient.
  • In between, the degree-2 curve follows the pattern and ignores most of the noise.
Try it

Move one patient and watch the curve

Choose a model. Then drag any patient up or down. The straight line hardly moves. The degree-9 curve bends to chase the patient you move: it is fitting the noise.

Look at the training error as you change the model: it is lowest for degree 9. A more flexible model always fits its training patients at least as well as a simpler one. So the training error cannot tell you when to stop. The next section shows what can.

The same thing in Python
import numpy as np
x = np.array([0.4, 1, 2, 2.9, 3.2, 4.6, 5.2, 7.6, 7.8, 9.3, 9.6, 9.8])   # dose, mg
y = np.array([2.6, 8.6, 10.8, 19, 19.9, 22.9, 18.4, 24, 20.7, 25.1, 24.5, 23.3])
u = x / 5 - 1                     # rescale 0..10 mg to -1..1 (keeps high powers tame)
for d in [1, 2, 9]:
    coef = np.polyfit(u, y, d)    # least squares polynomial of degree d
    print(d, round(np.mean((y - np.polyval(coef, u)) ** 2), 1))
# 1 13.4  /  2 5.2  /  9 0.7     training error falls as the degree rises
Quick check

A model predicts HbA1c from 15 blood tests. On the 300 patients it was trained on, its error is 0.01. On the next 300 patients it is 1.8. What is the most likely problem?

Section 3

3 Training and test data

To see how a model does on new patients, keep some patients aside. Before fitting, split the data into two groups:

  • The training set: the patients the model learns from.
  • The test set: patients the model never sees while it learns. The error on them estimates how the model will do on new patients.
A plot of error against the degree of the polynomial, from 1 to 9. The training error falls steadily from 13.4 to 0.7. The test error falls from 13.8 to 7.0 at degree 2, then rises to 24.9 at degree 9. The lowest test error is at degree 2. Left of it the model underfits; right of it the model overfits.

As the degree goes up, the training error keeps falling. The test error falls at first, then rises again: it makes a U shape. The bottom of the U, here degree 2, is the model that will do best on new patients.

Try it

Training error falls, test error turns

Move the degree. The top plot shows the curve with the training patients (blue) and the test patients (orange). The bottom plot shows both errors for every degree, with your choice marked.

Validation and cross-validation

If you choose the degree by looking at the test error, the test set has helped to build the model, and it no longer gives a fair estimate. So in practice the data are split three ways: training, validation (to choose settings such as the degree) and test (used once, at the very end). With few patients, cross-validation splits the data into, say, 5 parts, and lets each part take a turn as the validation set.

Data leakage

The test patients must be truly new. If the same patient is in both sets, for example two clinic visits of one patient, or two biopsies from one tumour, the model has already seen them. The test error then looks better than it really is. This is called data leakage. Split the data by patient, not by sample.
The same thing in Python
import numpy as np
rng = np.random.default_rng(0)
dose = rng.uniform(0, 10, 24)                            # 24 made-up patients
fall = 30 * dose / (dose + 2) + rng.normal(0, 3, 24)
idx = rng.permutation(24); tr, te = idx[:12], idx[12:]   # hold out 12 patients
for d in [1, 2, 9]:
    c = np.polyfit(dose[tr] / 5 - 1, fall[tr], d)        # fit on the training patients only
    err = (fall - np.polyval(c, dose / 5 - 1)) ** 2
    print(d, round(err[tr].mean(), 1), round(err[te].mean(), 1))
# 1 22.6 26.0  /  2 5.4 9.9  /  9 0.3 352.0   training error, then test error
Quick check

Two models predict length of stay. Model A: training error 2.1, test error 2.3. Model B: training error 0.4, test error 6.5. Which model should the hospital use for new patients?

Section 4

4 Bias and variance

Imagine running the same study many times. Each time you recruit new patients from the same population and fit the same kind of model. Two different things can go wrong:

  • Bias: the model is wrong in the same way every time. A straight line always misses the bend at low doses. More patients do not fix this.
  • Variance: the model changes a lot when the training patients change. A degree-9 curve is right on average, but every study gives a very different curve.
Two panels with the true dose-response curve in grey. The study is repeated six times with twelve new patients each time. In the left panel the six straight lines are close together, but all of them miss the bend: high bias. In the right panel the six degree-9 curves follow the truth on average, but they wiggle in different ways: high variance.
Try it

Repeat the study: bias and variance

Press New study, or Run 20 studies. Each study draws new patients and fits both models. Compare how the two models change from study to study. Then try 192 patients per study.

Simple models have high bias and low variance. Flexible models have low bias and high variance. The error on new patients contains both, plus noise that no model can predict:

$$\text{expected error on new patients} = \text{bias}^2 + \text{variance} + \text{noise}$$Noise is the part no model can predict, such as the day-to-day changes in a patient's blood pressure.

This is the bias–variance trade-off. On the left of the U-curve, bias is the problem: the model underfits. On the right, variance is the problem: the model overfits. The best model balances the two.

More data reduces variance

With 12 patients per study, the degree-9 prediction at 1 mg changes by about 6 mmHg (one standard deviation) from study to study. With 192 patients per study it changes by less than 1 mmHg. The straight line's bias, about +2 to +3 mmHg, does not go away with more patients. More data helps a flexible model. It cannot rescue a model that is too simple.
The same thing in Python
import numpy as np
rng = np.random.default_rng(2)
dose = np.linspace(0.5, 9.5, 12)
for d in [1, 9]:
    pred = []
    for study in range(1000):                    # repeat the study 1,000 times
        fall = 30 * dose / (dose + 2) + rng.normal(0, 3, 12)
        c = np.polyfit(dose / 5 - 1, fall, d)
        pred.append(np.polyval(c, 1 / 5 - 1))    # prediction at 1 mg
    print(d, round(np.mean(pred) - 10, 1), round(np.std(pred), 1))
# 1 2.4 1.5   line: about 2.4 too high every time (bias), but stable
# 9 0.1 6.2   degree 9: right on average (truth 10), but spread widely
Quick check

A hospital retrains a risk model every month on that month's patients. Each month the model gives very different predictions for the same kind of patient. What does the model have, and what would help?

Section 5

5 Regularisation: a penalty for large weights

Choosing a lower degree is one way to stop overfitting. Another way keeps the flexible model, but makes large weights costly. Overfit models usually have huge weights. The degree-9 fit in this lesson has a weight of more than 1,000: large positive and negative terms almost cancel, and what is left over makes the wiggles. Regularisation adds a penalty for large weights to the loss:

$$\text{loss} = \text{MSE} + \lambda \times (\text{size of the weights})$$λ (lambda) sets how strong the penalty is. With λ = 0 this is the plain MSE.
The degree-9 model fitted to the twelve patients with a growing penalty. With lambda 0 the largest weight is 1052, the curve is wild and the test error is 24.9. With lambda 0.0001 the largest weight is 37 and the test error 10.6. With lambda 0.03 the curve is smooth, the largest weight is 4.9 and the test error 9.1. With lambda 10 the curve is almost flat and the test error is 58.6: the model now underfits.

Now the optimiser has two goals: fit the patients (a small MSE) and keep the weights small (a small penalty). λ sets the balance:

  • \(\lambda = 0\): no penalty. The degree-9 model overfits. Test error 24.9.
  • A moderate \(\lambda\) (0.03 here): the curve is smooth. Test error 9.1.
  • A very large \(\lambda\) (10): every weight is pushed almost to 0. The curve goes flat, and the model underfits. Test error 58.6.

There are two common ways to measure the "size of the weights":

NamePenaltyWhat it does
L2, or ridge\(\lambda \sum_j w_j^2\)Shrinks all the weights smoothly towards 0. None of them becomes exactly 0.
L1, or lasso\(\lambda \sum_j |w_j|\)Shrinks the weights, and sets some of them exactly to 0. The model then ignores those inputs.
Try it

Ridge and lasso on the degree-9 model

Move λ from 0 upwards and watch the curve calm down and the weights shrink (the bars use a log scale). Then switch to lasso: weights drop to exactly 0 one by one. Push λ to the end: the curve goes flat.

Why lasso matters in precision health. A gene-expression study may measure 20,000 genes in 200 patients. A linear model with one weight for each gene has far more weights than patients. Least squares then has no single answer: there are endless ways to fit the 200 patients perfectly, and all of them overfit. Lasso keeps a small set of genes and sets the weights of all the others to exactly zero. The result is a short list of genes that can be checked in the laboratory: a gene signature.

Why you will care later

λ is a hyperparameter, like the learning rate. It is usually chosen by cross-validation. You will meet ridge and lasso, and the elastic net that mixes them, in regression and genomics. Neural networks use the same L2 penalty under the name weight decay.
The same thing in Python
import numpy as np
x = np.array([0.4, 1, 2, 2.9, 3.2, 4.6, 5.2, 7.6, 7.8, 9.3, 9.6, 9.8])   # dose, mg
y = np.array([2.6, 8.6, 10.8, 19, 19.9, 22.9, 18.4, 24, 20.7, 25.1, 24.5, 23.3])
X = np.vander(x / 5 - 1, 10, increasing=True)[:, 1:]   # powers 1 to 9
Xc, yc = X - X.mean(0), y - y.mean()       # centre: the intercept is not penalised
n, p = X.shape
for lam in [0, 0.0001, 0.03, 10]:          # ridge: minimise MSE + lam * sum(w**2)
    w = np.linalg.solve(Xc.T @ Xc / n + lam * np.eye(p), Xc.T @ yc / n)
    print(lam, round(np.abs(w).max(), 1))  # the largest weight
# 0 1052.0  /  0.0001 37.2  /  0.03 4.9  /  10 0.4
Quick check

You fit a lasso model to the expression of 5,000 genes. What happens as you make λ larger?

Section 6

6 Where you go from here

This is the end of the course. You now have the mathematics that statistics and machine learning are built on. Here is a map from each unit to where you will meet it again.

UnitYou learnedWhere it comes back
1 Functions and notation\(\Sigma\) and \(\Pi\), lines, exponentials, logs, odds, the sigmoidReading any formula in a paper; log fold changes in genomics; odds ratios; logistic regression
2 Linear algebraVectors, matrices, \(\hat{y} = X\beta\)Multiple regression; principal component analysis (PCA); every layer of a neural network
3 CalculusDerivatives, partial derivatives, gradients, areas under curvesTraining any model by gradient descent; drug concentration over time; probabilities as areas
4 ProbabilityConditional probability, Bayes' theorem, distributionsDiagnostic tests and screening; Bayesian statistics; simulation
5 Statistical foundationsSamples, standard error, correlation, likelihood, least squaresConfidence intervals and p-values; fitting linear, logistic and survival models
6 OptimisationLoss functions, gradient descent, overfitting, regularisationMachine learning: cross-validation, lasso gene signatures, neural networks

Methods you are now ready to learn:

  • Linear regression
  • Logistic regression
  • Survival analysis and Cox models
  • Principal component analysis
  • Clustering
  • Ridge, lasso and elastic net
  • Cross-validation
  • Decision trees and random forests
  • Neural networks
  • Polygenic risk scores
  • Bayesian statistics

Four questions for any model you meet

  1. What is the model? Which inputs does it use, and what are its weights?
  2. What is the loss? MSE, log loss, or something else?
  3. How are the weights found? A direct formula, or gradient descent?
  4. How was it tested? On patients it had never seen, and without leakage?

You do not need to remember every formula from this course. When you meet one, read it from the inside out, as in lesson 1.1, and ask these four questions. That is what working statisticians and machine-learning scientists do every day. Well done for reaching the end.

Section 7

7 Check your understanding

Five short questions on this lesson. Nothing is graded or saved outside this browser. If you choose a wrong option you get a hint, and you can try again.

Check your understanding

A straight-line model has a training error of 25 and a test error of 26. A curve that can bend has errors of 6 and 7. What was wrong with the straight line?

Check your understanding

A study has 3 skin images from each of 500 patients. The images are shuffled and split at random into a training set and a test set. What is the problem?

Check your understanding

The test errors for polynomial degrees 1 to 6 are 14, 7, 9, 11, 12 and 16. Which degree would you choose for new patients?

Check your understanding

Which sentence describes high variance?

Check your understanding

A ridge model is fitted with a very large λ. What do you expect?