Bias–variance & CV formulas

Bias–variance trade-off and cross-validation. 15 formulas for GATE DA Machine Learning, each with when to use it. Each is explained in the topic notes.

All 15 Bias–variance & CV formulas

BIAS OF AN ESTIMATOR AT $X_0$

Bias (\hat{f} (x_{0})) = E [\hat{f} (x_{0})] - f (x_{0})

Systematic error: how far the average prediction over training sets is from the truth.

VARIANCE OF AN ESTIMATOR AT $X_0$

Var (\hat{f} (x_{0})) = E [(\hat{f} (x_{0}) - E [\hat{f} (x_{0})])^{2}]

Sensitivity: how much the prediction changes from one training set to another.

BIAS-VARIANCE DECOMPOSITION OF EXPECTED SQUARED ERROR

E [(y - \hat{f} (x_{0}))^{2}] = Bias^{2} (\hat{f} (x_{0})) + Var (\hat{f} (x_{0})) + σ^{2}

Valid for squared-error loss, with zero-mean noise in the test response that is independent of the training data. The term $σ^{2}$ is irreducible: no model can remove it.

RIDGE REGRESSION ESTIMATE

\hat{w}_{ridge} = ar g w min ∥ y - X w ∥^{2} + λ ∥ w ∥^{2} = (X^{⊤} X + λ I)^{- 1} X^{⊤} y

Convention: $X$ is $n \times p$ with samples in its rows, $y$ holds the $n$ responses, and $w$ the weights. The bias (intercept) $b$ is not penalised: $y$ and each column of $X$ are centred first, so $b$ drops out and is recovered afterwards. For $λ > 0$ , $X^{⊤} X + λ I$ is always invertible. $λ = 0$ gives ordinary least squares (when $X^{⊤} X$ is invertible), the lowest-bias and highest-variance member of the ridge family. $λ \to \infty$ drives $\hat{w} \to 0$ (high bias, low variance). The penalty depends on the scale of each feature, so features are usually standardised first.

EXPECTED SQUARED ERROR OF $K$-NN REGRESSION AT $X_0$

E [(y - \hat{f}_{k} (x_{0}))^{2}] = σ^{2} + [f (x_{0}) - \frac{1}{k} l = 1 \sum k f (x_{(l)})]^{2} + \frac{σ ^{2}}{k}

Holds when the training inputs are fixed and $y_{i} = f (x_{i}) + ε_{i}$ with independent zero-mean noise of variance $σ^{2}$ . Here $x_{(1)}, \dots, x_{(k)}$ are the $k$ training inputs nearest to $x_{0}$ , and $\hat{f}_{k} (x_{0})$ is the average of their $y$ values. The middle term is bias squared, which usually grows with $k$ as farther neighbours are averaged in. The variance $σ^{2} / k$ falls as $k$ grows.

ERROR ON ONE FOLD (REGRESSION, SQUARED ERROR)

E_{i} = \frac{1}{n _{i}} j \in fold i \sum (y_{j} - \overset{y}{^}_{j})^{2}

$n_{i}$ is the number of samples in fold $i$ , and $\overset{y}{^}_{j}$ is the prediction of the model trained without fold $i$ . For classification, $E_{i}$ is the misclassification rate on fold $i$ .

$K$-FOLD CV ESTIMATE

CV_{(k)} = \frac{1}{k} i = 1 \sum k E_{i}

Assumes equal-sized folds. With unequal folds, use $\sum_{i = 1}^{k} (n_{i} / n) E_{i}$ . Number of models trained: $k$ , each on about $n (k - 1) / k$ samples.

LEAVE-ONE-OUT (LOO) CV ESTIMATE

CV_{(n)} = \frac{1}{n} i = 1 \sum n ℓ (y_{i}, \hat{f}^{(- i)} (x_{i}))

LOO is $k$ -fold with $k = n$ . $ℓ$ is the loss, and $\hat{f}^{(- i)}$ is trained on every sample except the $i$ -th, so $n$ models are trained, each on $n - 1$ samples.

LOO SHORTCUT FOR ORDINARY LEAST SQUARES

CV_{(n)} = \frac{1}{n} i = 1 \sum n (\frac{y _{i} - y ^ _{i}}{1 - h _{ii}})^{2}

Valid only for linear least squares with squared error, fitted once on all the data. $\overset{y}{^}_{i}$ is the full-data fit. $h_{ii} = x_{i}^{⊤} (X^{⊤} X)^{- 1} x_{i}$ is the $i$ -th diagonal entry of the hat matrix $H = X (X^{⊤} X)^{- 1} X^{⊤}$ , with samples in the rows of $X$ . It requires $X^{⊤} X$ to be invertible and $h_{ii} < 1$ . If an intercept is fitted, $X$ includes a column of ones. No refitting is needed.

MODELS TRAINED IN A GRID SEARCH

N_{models} = (number of candidate settings) \times k + 1 (final refit)

With several hyperparameters, the number of candidate settings is the product of the grid sizes. The $+ 1$ is the refit on all the data. Leave it out if the question asks only about the search.

ACCURACY

Accuracy = \frac{T P + T N}{T P + F P + F N + T N}

Misleading when the classes are very unbalanced.

PRECISION

Precision = \frac{T P}{T P + F P}

Of everything predicted positive, the fraction that is truly positive. Undefined if nothing is predicted positive.

RECALL (SENSITIVITY, TRUE POSITIVE RATE)

Recall = \frac{T P}{T P + F N}

Of everything truly positive, the fraction found.

SPECIFICITY AND FALSE POSITIVE RATE

Specificity = \frac{T N}{T N + F P}, FPR = \frac{F P}{F P + T N} = 1 - Specificity

Specificity is the fraction of true negatives correctly identified.

F1 SCORE

F_{1} = \frac{2 P R}{P + R} = \frac{2 T P}{2 T P + F P + F N}

The harmonic mean of precision $P$ and recall $R$ . TN does not appear in it.

Where this was asked: 2026 (2). See the 2 previous-year questions on Bias–variance & CV. This shows what was asked, not what will be asked.

KEEP GOING

Bias–variance & CV: the notes and 2 solved previous-year questions