GATE DA

Bias–variance trade-off and cross-validation

Every model is judged by how well it does on data it has not seen, not on the data it was fitted to. This topic explains why training error misleads, how expected test error splits into bias, variance and noise, how cross-validation (k-fold and leave-one-out) estimates generalisation error and picks hyperparameters, and the common measures used to score a classifier.

2 previous-year questions · 2 solved · 7 practice questions

Training, validation and test error

A model is fitted on a training set. Its error there is the training error. What we actually care about is the generalisation error: the expected error on new data from the same distribution. We estimate it on data the model never saw during fitting.

Set	Used for	Can it be reused to tune the model?
Training	Fitting the parameters (weights, splits, centroids)	Yes, this is its purpose
Validation	Choosing hyperparameters and comparing models	Yes, but the more choices made on it, the more optimistic its error becomes
Test	One final, honest estimate of generalisation error	No. Using it for any choice spoils the estimate

Training error is usually lower than test error, because the parameters were chosen to make it small. Training error alone therefore cannot tell you whether a model is good. A very flexible model can reach zero training error and still predict badly.

EXAM TIP

A hyperparameter (the $k$ in $k$ -NN, the penalty $λ$ , the tree depth, the polynomial degree) is not learned by minimising training error. Choosing it that way tends to pick the most flexible setting. It must be chosen on validation data or by cross-validation.

Underfitting and overfitting

Underfitting: the model is too simple to capture the true pattern. Training error is high, and test error is high and close to the training error. Overfitting: the model is so flexible that it fits the noise in the training set. Training error is very low, test error is much higher, and the gap between them is large.

Symptom	Training error	Validation/test error	Gap	Diagnosis
Both high	High	High	Small	Underfitting (high bias)
Low train, high test	Low	High	Large	Overfitting (high variance)
Both low	Low	Low	Small	Good fit

Typical remedies

Underfitting: use a more flexible model, add useful features, reduce regularisation. More data of the same kind does not cure underfitting.
Overfitting: get more training data, use a simpler model, increase regularisation, remove irrelevant features, stop training earlier.

EXAM TIP

As model complexity increases, training error generally falls. For nested models fitted by least squares it can never rise. Validation error is typically U-shaped: it falls, reaches a minimum, then rises. The best model sits near the bottom of the U.

The bias-variance decomposition

Suppose $y = f (x) + ε$ , where $ε$ is noise with $E [ε] = 0$ and $Var (ε) = σ^{2}$ . Let $\hat{f}$ be the model learned from a random training set. At a fixed point $x_{0}$ , let $y$ be a new observation whose noise is independent of the training set. The expectation is over the random training set and the noise in this new $y$ . Squared-error loss is assumed throughout.

BIAS OF AN ESTIMATOR AT $X_0$

Bias (\hat{f} (x_{0})) = E [\hat{f} (x_{0})] - f (x_{0})

Systematic error: how far the average prediction over training sets is from the truth.

VARIANCE OF AN ESTIMATOR AT $X_0$

Var (\hat{f} (x_{0})) = E [(\hat{f} (x_{0}) - E [\hat{f} (x_{0})])^{2}]

Sensitivity: how much the prediction changes from one training set to another.

BIAS-VARIANCE DECOMPOSITION OF EXPECTED SQUARED ERROR

E [(y - \hat{f} (x_{0}))^{2}] = Bias^{2} (\hat{f} (x_{0})) + Var (\hat{f} (x_{0})) + σ^{2}

Valid for squared-error loss, with zero-mean noise in the test response that is independent of the training data. The term $σ^{2}$ is irreducible: no model can remove it.

For an estimator $\hat{θ}$ of a fixed parameter $θ$ , the same idea gives the mean squared error: $MSE (\hat{θ}) = E [(\hat{θ} - θ)^{2}] = Bias^{2} (\hat{θ}) + Var (\hat{θ})$ .

EXAM TIP

The decomposition has bias squared, not bias. A bias of $- 0.5$ contributes $0.25$ . All three terms are non-negative, so under these assumptions the expected test error at $x_{0}$ can never be below $σ^{2}$ .

What moves bias and variance

The trade-off: making a model more flexible usually lowers bias and raises variance. Making it simpler does the opposite. Test error is smallest where the two balance. The table shows the usual direction of each change, not a guarantee for every dataset.

Change	Bias	Variance
Increase model complexity (higher polynomial degree, deeper tree, more hidden units)	Decreases	Increases
Increase regularisation strength $λ$ (ridge, lasso, weight decay)	Increases	Decreases
Increase $k$ in $k$ -NN	Increases	Decreases
More training data (same model class)	Roughly unchanged	Decreases
More irrelevant features	Little change	Increases
Averaging many models trained on resamples (bagging)	Roughly unchanged	Decreases

Two points to remember. First, more data helps against variance, not against bias: a straight line fitted to curved data stays wrong however many points you give it. Second, regularisation deliberately accepts a small bias to buy a larger drop in variance.

RIDGE REGRESSION ESTIMATE

\hat{w}_{ridge} = ar g w min ∥ y - X w ∥^{2} + λ ∥ w ∥^{2} = (X^{⊤} X + λ I)^{- 1} X^{⊤} y

Convention: $X$ is $n \times p$ with samples in its rows, $y$ holds the $n$ responses, and $w$ the weights. The bias (intercept) $b$ is not penalised: $y$ and each column of $X$ are centred first, so $b$ drops out and is recovered afterwards. For $λ > 0$ , $X^{⊤} X + λ I$ is always invertible. $λ = 0$ gives ordinary least squares (when $X^{⊤} X$ is invertible), the lowest-bias and highest-variance member of the ridge family. $λ \to \infty$ drives $\hat{w} \to 0$ (high bias, low variance). The penalty depends on the scale of each feature, so features are usually standardised first.

EXPECTED SQUARED ERROR OF $K$-NN REGRESSION AT $X_0$

E [(y - \hat{f}_{k} (x_{0}))^{2}] = σ^{2} + [f (x_{0}) - \frac{1}{k} l = 1 \sum k f (x_{(l)})]^{2} + \frac{σ ^{2}}{k}

Holds when the training inputs are fixed and $y_{i} = f (x_{i}) + ε_{i}$ with independent zero-mean noise of variance $σ^{2}$ . Here $x_{(1)}, \dots, x_{(k)}$ are the $k$ training inputs nearest to $x_{0}$ , and $\hat{f}_{k} (x_{0})$ is the average of their $y$ values. The middle term is bias squared, which usually grows with $k$ as farther neighbours are averaged in. The variance $σ^{2} / k$ falls as $k$ grows.

EXAM TIP

$k$ -NN with $k = 1$ has zero training error when all training inputs are distinct (each point is its own nearest neighbour), and it has high variance. Do not confuse a small $k$ with a simple model: small $k$ is the flexible end.

Cross-validation: k-fold and leave-one-out

The simplest estimate is the hold-out (validation-set) method: split the data once, train on one part and measure the error on the other. It has two weaknesses. The estimate depends on which samples happen to land in the validation part, and the model trains on fewer samples, so its error tends to be pessimistic. Cross-validation (CV) reuses every sample for both training and validation, in turn, so that each sample is validated exactly once by a model that did not train on it.

$k$-fold cross-validation, in order

Shuffle the $n$ samples and split them into $k$ folds of (nearly) equal size.
For $i = 1, \dots, k$ : train a model on the other $k - 1$ folds and compute its error $E_{i}$ on fold $i$ .
Report the average of $E_{1}, \dots, E_{k}$ as the CV estimate.

ERROR ON ONE FOLD (REGRESSION, SQUARED ERROR)

E_{i} = \frac{1}{n _{i}} j \in fold i \sum (y_{j} - \overset{y}{^}_{j})^{2}

$n_{i}$ is the number of samples in fold $i$ , and $\overset{y}{^}_{j}$ is the prediction of the model trained without fold $i$ . For classification, $E_{i}$ is the misclassification rate on fold $i$ .

$K$-FOLD CV ESTIMATE

CV_{(k)} = \frac{1}{k} i = 1 \sum k E_{i}

Assumes equal-sized folds. With unequal folds, use $\sum_{i = 1}^{k} (n_{i} / n) E_{i}$ . Number of models trained: $k$ , each on about $n (k - 1) / k$ samples.

LEAVE-ONE-OUT (LOO) CV ESTIMATE

CV_{(n)} = \frac{1}{n} i = 1 \sum n ℓ (y_{i}, \hat{f}^{(- i)} (x_{i}))

LOO is $k$ -fold with $k = n$ . $ℓ$ is the loss, and $\hat{f}^{(- i)}$ is trained on every sample except the $i$ -th, so $n$ models are trained, each on $n - 1$ samples.

LOO SHORTCUT FOR ORDINARY LEAST SQUARES

CV_{(n)} = \frac{1}{n} i = 1 \sum n (\frac{y _{i} - y ^ _{i}}{1 - h _{ii}})^{2}

Valid only for linear least squares with squared error, fitted once on all the data. $\overset{y}{^}_{i}$ is the full-data fit. $h_{ii} = x_{i}^{⊤} (X^{⊤} X)^{- 1} x_{i}$ is the $i$ -th diagonal entry of the hat matrix $H = X (X^{⊤} X)^{- 1} X^{⊤}$ , with samples in the rows of $X$ . It requires $X^{⊤} X$ to be invertible and $h_{ii} < 1$ . If an intercept is fitted, $X$ includes a column of ones. No refitting is needed.

Property	$k$ -fold ( $k$ = 5 or 10)	Leave-one-out ( $k = n$ )
Models trained	$k$	$n$
Training size per model	$n (k - 1) / k$	$n - 1$
Cost	Lower	Highest (unless a shortcut exists)
Bias of the error estimate	Slightly pessimistic (models see less data)	Smallest (models see almost all data)
Variance of the estimate	Usually lower	Can be higher, since the $n$ training sets overlap almost completely
Randomness	Depends on how folds are drawn	None: the splits are fixed

EXAM TIP

Cross-validation estimates how well the procedure generalises. The final model is normally refit on all the data afterwards, and the $k$ models inside CV are not the model you deploy. Any preprocessing learned from data (scaling constants, feature selection) must be fitted inside each training fold only. Fitting it on all the data first leaks information and makes the CV error optimistic. For classification, stratified folds keep the class proportions in each fold.

Choosing a hyperparameter by cross-validation

Procedure

List candidate values of the hyperparameter (for example $λ \in {0.01, 0.1, 1, 10}$ ).
For each candidate, run $k$ -fold CV and record the CV error.
Pick the candidate with the smallest CV error.
Refit the model with that value on the whole training set.
Evaluate once on a separate test set, if one was held out.

MODELS TRAINED IN A GRID SEARCH

N_{models} = (number of candidate settings) \times k + 1 (final refit)

With several hyperparameters, the number of candidate settings is the product of the grid sizes. The $+ 1$ is the refit on all the data. Leave it out if the question asks only about the search.

WORKED EXAMPLE

A dataset has $n = 120$ samples. 5-fold CV is used and the fold errors (misclassification rates) are $0.10, 0.14, 0.12, 0.16, 0.08$ . Find the CV estimate and the size of each training set.

Solution

Each fold has $120/5 = 24$ samples, so the folds are equal in size.
Each model trains on the other four folds: $4 \times 24 = 96$ samples.
CV estimate $= (0.10 + 0.14 + 0.12 + 0.16 + 0.08) /5 = 0.60/5 = 0.12$ .

Answer: CV estimate $= 0.12$ ; each of the 5 models is trained on 96 samples.

EXAM TIP

If the hyperparameter is chosen by CV and the same CV error is then quoted as the model's performance, the figure is optimistic, because the winner was picked for having the lowest error. An untouched test set gives the honest number. Without one, nested cross-validation can be used: an inner CV picks the hyperparameter inside each outer training fold, and the outer folds measure the error.

Evaluation measures for classification

For a binary problem with a positive class, the confusion matrix counts: TP (positive predicted positive), FP (negative predicted positive), FN (positive predicted negative), TN (negative predicted negative). Here $n = T P + F P + F N + T N$ .

ACCURACY

Accuracy = \frac{T P + T N}{T P + F P + F N + T N}

Misleading when the classes are very unbalanced.

PRECISION

Precision = \frac{T P}{T P + F P}

Of everything predicted positive, the fraction that is truly positive. Undefined if nothing is predicted positive.

RECALL (SENSITIVITY, TRUE POSITIVE RATE)

Recall = \frac{T P}{T P + F N}

Of everything truly positive, the fraction found.

SPECIFICITY AND FALSE POSITIVE RATE

Specificity = \frac{T N}{T N + F P}, FPR = \frac{F P}{F P + T N} = 1 - Specificity

Specificity is the fraction of true negatives correctly identified.

F1 SCORE

F_{1} = \frac{2 P R}{P + R} = \frac{2 T P}{2 T P + F P + F N}

The harmonic mean of precision $P$ and recall $R$ . TN does not appear in it.

WORKED EXAMPLE

A classifier is tested on 200 samples and gives $T P = 40$ , $F P = 10$ , $F N = 20$ , $T N = 130$ . Find the accuracy, precision, recall and $F_{1}$ .

Solution

Accuracy $= (40 + 130) /200 = 170/200 = 0.85$ .
Precision $= 40/ (40 + 10) = 0.80$ .
Recall $= 40/ (40 + 20) = 2/3 \approx 0.667$ .
$F_{1} = 2 \cdot 40/ (2 \cdot 40 + 10 + 20) = 80/110 \approx 0.727$ .

Answer: Accuracy 0.85, precision 0.80, recall 0.667, $F_{1} \approx 0.727$ .

EXAM TIP

A model that always predicts the majority class can have high accuracy and zero recall for the minority class. When the positive class is rare, quote precision, recall or $F_{1}$ , not accuracy alone.

PREVIOUS-YEAR QUESTIONS

2 questions from the DA papers

Try each question first, then open the answer. We placed each question under a topic ourselves; the exam paper does not label topics.

See these papers in full: GATE DA 2026

GATE 2026 DA Q.12

GATE 2026 · DAQ.12 · 1 mark · MCQ

Consider that you are training a classifier for a 10-class classification problem. Each input is represented as a 512-dimensional vector. There are 1000 samples, out of which first 100 will be used for testing. Let Leave-One-Out-Cross-Validation (LOOCV) be used for selection of the classifier model before testing.

Which of the following options is the correct number of validation splits that will be generated?

(A)
10
(B)
512
(C)
900
(D)
1000

Answer and solution

GATE 2026 DA Q.12: answer and solution

See the question in the paper

Tap to open full size

GATE 2026 DA Q.12: answer and solution

GATE 2026 DA Q.47

GATE 2026 · DAQ.47 · 2 marks · MSQ

Consider that 20 stories of Author X and 10 stories of Author Y were kept together without mentioning the names of the authors. A classifier was then asked to predict the author (X or Y) of each of these stories. Let, out of X's stories, 6 were classified as that of Y. On the other hand, out of Y's stories, 2 were classified as that of X.

Considering X and Y as two classes, which of the following statements is/are true?

(A)
Accuracy of the classifier is 11/15.
(B)
Precision of Class X is higher than the Precision of Class Y.
(C)
Recall of Class X is higher than the Recall of Class Y.
(D)
Accuracy of the classifier is 14/15.

Answer and solution

GATE 2026 DA Q.47: answer and solution

See the question in the paper

Tap to open full size

GATE 2026 DA Q.47: answer and solution

PRACTICE

7 questions to test yourself

PRACTICE 11 mark · NAT

At a test point $x_{0}$ the true value is $f (x_{0}) = 3.0$ . Over many training sets, a model's prediction at $x_{0}$ has mean $3.2$ and variance $0.09$ . The noise in $y$ has zero mean and variance $σ^{2} = 0.25$ , and is independent of the training set. Using squared-error loss, find the expected squared prediction error at $x_{0}$ (answer to two decimal places).

Answer and solution

Answer: $0.38$ (accepted range $\pm 0.005$ )

Bias $= E [\hat{f} (x_{0})] - f (x_{0}) = 3.2 - 3.0 = 0.2$ , so $Bias^{2} = 0.04$ .
Variance $= 0.09$ and irreducible noise $= 0.25$ .
Expected error $= 0.04 + 0.09 + 0.25 = 0.38$ .

PRACTICE 21 mark · MCQ

In ridge regression, the penalty strength $λ$ is increased from a small positive value to a much larger one, with the training set fixed. Which statement describes the typical effect on the fitted model?

(A)
Bias decreases and variance increases
(B)
Bias increases and variance decreases
(C)
Both bias and variance decrease
(D)
Both bias and variance increase

Answer and solution

Answer: (B)

A larger $λ$ shrinks the weights towards zero, which makes the model less flexible.
A less flexible model is less sensitive to the particular training set, so variance typically falls.
It also cannot fit the true pattern as closely, so bias typically rises. The answer is B.

PRACTICE 32 marks · MSQ

A dataset has $n = 100$ samples. Which of the following statements about leave-one-out (LOO) cross-validation and 5-fold cross-validation on this dataset are correct?

(A)
LOO trains 100 models.
(B)
Each LOO model is trained on 99 samples.
(C)
5-fold CV trains 5 models, each on 80 samples.
(D)
LOO always needs less computation than 5-fold CV.

Answer and solution

Answer: (A), (B), (C)

LOO leaves out one sample each time, so it trains $n = 100$ models, each on $n - 1 = 99$ samples. A and B are true.
5-fold CV has 5 folds of 20 samples. Each model trains on the other 4 folds, which is $80$ samples. C is true.
LOO trains 100 models against 5, so it normally costs more, not less. D is false: a shortcut such as the least-squares formula is a special case, not 'always'.

PRACTICE 42 marks · MCQ

A decision tree reaches a training error of 1% but a validation error of 22%. Which diagnosis and remedy fit best?

(A)
High bias; use a more complex model
(B)
High variance (overfitting); more training data or stronger regularisation, such as limiting tree depth, is likely to help
(C)
High bias; reduce regularisation
(D)
Neither; a gap this size is expected for any model with noise

Answer and solution

Answer: (B)

Training error is very low but validation error is much higher, so the gap is large.
A large gap means overfitting, which is high variance.
The standard remedies are more data, a simpler model or more regularisation. A and C treat it as underfitting, which would show high training error. The answer is B.

PRACTICE 51 mark · NAT

A model has two hyperparameters: the first has 4 candidate values and the second has 3. Every combination is evaluated by 5-fold cross-validation, and the best combination is then refit once on the full training set. How many times is a model trained in total?

Answer and solution

Answer: $61$

Number of candidate settings $= 4 \times 3 = 12$ .
Each setting needs 5 models (one per fold): $12 \times 5 = 60$ .
Add the single final refit: $60 + 1 = 61$ .

PRACTICE 62 marks · NAT

A $k$ -NN regressor with $k = 4$ is used at a point $x_{0}$ . The training inputs are fixed, and each response is $y_{i} = f (x_{i}) + ε_{i}$ with independent zero-mean noise of variance $σ^{2} = 0.8$ . The true function values at the 4 nearest training inputs are $2.0, 2.4, 2.2, 2.6$ , and $f (x_{0}) = 2.5$ . Using squared-error loss, find the expected squared prediction error at $x_{0}$ for a new observation $y$ at $x_{0}$ (answer to two decimal places).

Answer and solution

Answer: $1.04$ (accepted range $\pm 0.005$ )

Average of the neighbours' true values $= (2.0 + 2.4 + 2.2 + 2.6) /4 = 9.2/4 = 2.3$ .
Bias $= 2.3 - 2.5 = - 0.2$ , so $Bias^{2} = 0.04$ .
Variance of the prediction $= σ^{2} / k = 0.8/4 = 0.2$ .
Expected error $= σ^{2} + Bias^{2} + σ^{2} / k = 0.8 + 0.04 + 0.2 = 1.04$ .

PRACTICE 72 marks · NAT

A binary classifier is evaluated on 100 samples, giving $T P = 30$ , $F P = 10$ , $F N = 20$ and $T N = 40$ . Find the $F_{1}$ score of the positive class, rounded to two decimal places.

Answer and solution

Answer: $0.67$ (accepted range $\pm 0.01$ )

Precision $= 30/ (30 + 10) = 0.75$ .
Recall $= 30/ (30 + 20) = 0.60$ .
$F_{1} = 2 \cdot 30/ (2 \cdot 30 + 10 + 20) = 60/90 \approx 0.67$ .

OFFICIAL GATE 2027 SYLLABUS WORDING

Copied from the GATE 2027 syllabus; see all 2027 syllabi.

GATE DA

bias-variance trade-off, cross-validation methods such as leave-one-out (LOO) cross-validation, k-folds cross-validation

Preparing in Kolkata? IAE Academy's course for GATE DA.