GATE DA

Dimensionality reduction and principal component analysis

Dimensionality reduction replaces many correlated features by a few new ones that keep as much of the data's variance as possible. Principal component analysis does this with the eigenvectors and eigenvalues of the covariance matrix, or equivalently the SVD of the centred data. GATE DA questions ask you to compute eigenvalues, the proportion of variance explained, a projected coordinate or a reconstruction error, or to judge what PCA can and cannot do.

2 previous-year questions · 2 solved · 5 practice questions

Why reduce the dimension

A data set with $d$ features lives in $R^{d}$ . When many features are correlated or nearly constant, the points really occupy a much smaller region of that space. Dimensionality reduction maps each point to $k < d$ numbers while losing as little information as possible.

Reasons to reduce

Fewer features mean fewer parameters, so models tend to overfit less when samples are few compared with $d$ .
Distances become less informative in very high dimensions; with fewer dimensions, nearest-neighbour and clustering methods can behave better.
Storage and computation fall.
Two or three components can be plotted, which helps to see structure.
With PCA, correlated features are replaced by uncorrelated ones.

Convention used here. Samples are the rows of the data matrix $X \in R^{n \times d}$ , so $x_{i}^{⊤}$ is row $i$ . Vectors are bold and are column vectors. Distances are Euclidean. Principal directions are taken as unit vectors, and eigenvalues are listed in decreasing order. The sample covariance uses the divisor $n - 1$ unless stated otherwise.

Feature selection keeps a subset of the original features. Feature extraction builds new features from all of them. PCA is feature extraction. It is unsupervised and linear: it never looks at labels, and each new feature is a linear combination of the centred original features.

Centring and scaling the data

PCA looks for directions of maximum variance about the mean, so the data must be centred first. Without centring, and when the mean is far from the origin compared with the spread, the first direction mostly points from the origin to the data cloud rather than along its spread.

SAMPLE MEAN AND CENTRED DATA

\overset{ˉ}{x} = \frac{1}{n} i = 1 \sum n x_{i}, x_{i}^{c} = x_{i} - \overset{ˉ}{x}

Centre each feature (column) to mean zero. $X_{c}$ denotes the centred matrix, whose rows are $(x_{i}^{c})^{⊤}$ .

SAMPLE COVARIANCE MATRIX

S = \frac{1}{n - 1} X_{c}^{⊤} X_{c} \in R^{d \times d}

Symmetric and positive semi-definite. Diagonal entries are feature variances; off-diagonal entries are covariances. Some texts divide by $n$ instead. The eigenvectors and the proportions of variance do not change; every eigenvalue is scaled by the same factor.

When to standardise. Variance depends on units. If one feature is measured in thousands and another in fractions, the first dominates the variance and PCA simply returns it. When features have different units or very different scales, divide each centred column by its standard deviation (a z-score) before PCA. If the standard deviation uses the same divisor as the covariance (here $n - 1$ ), $S$ becomes the correlation matrix $R$ . $R$ has 1 on every diagonal entry, so the total variance equals $d$ . When all features share the same units and the scale itself carries meaning, centring alone is usually used.

EXAM TIP

Changing the scale of one feature generally changes the principal directions of the covariance matrix, so PCA on $S$ and PCA on $R$ generally give different components. The exception is uncorrelated features: then $S$ is diagonal and the directions stay the coordinate axes (only their order can change).

Principal components from the covariance matrix

The first principal direction is the unit vector $v_{1}$ that maximises the variance of the projections $z_{i} = v^{⊤} x_{i}^{c}$ . For a unit vector $v$ , the variance of the projections is $v^{⊤} S v$ . Maximise $v^{⊤} S v$ subject to $∥ v ∥ = 1$ with a Lagrange multiplier. The condition is $S v = λ v$ , so the best $v$ is an eigenvector and the variance it reaches is its eigenvalue. The second direction maximises variance among unit vectors orthogonal to the first, and so on.

PRINCIPAL DIRECTIONS

S v_{j} = λ_{j} v_{j}, λ_{1} \geq λ_{2} \geq \dots \geq λ_{d} \geq 0, v_{i}^{⊤} v_{j} = 0 (i \neq = j)

The eigenvectors of the symmetric matrix $S$ are chosen orthonormal. $v_{j}$ is the $j$ -th principal direction (component). A direction is fixed only up to sign.

SPECTRAL DECOMPOSITION AND COVARIANCE OF SCORES

S = V Λ V^{⊤} = j = 1 \sum d λ_{j} v_{j} v_{j}^{⊤}, \frac{1}{n - 1} Z^{⊤} Z = Λ_{k}

$V = [v_{1} \dots v_{d}]$ is orthogonal and $Λ = diag (λ_{1}, \dots, λ_{d})$ . The scores $Z = X_{c} V_{k}$ (defined below) have the diagonal covariance $Λ_{k} = diag (λ_{1}, \dots, λ_{k})$ .

VARIANCE ALONG A PRINCIPAL DIRECTION

Var (v_{j}^{⊤} x^{c}) = v_{j}^{⊤} S v_{j} = λ_{j}

Projections onto different components are uncorrelated: $v_{i}^{⊤} S v_{j} = 0$ for $i \neq = j$ .

VARIANCE ALONG ANY UNIT DIRECTION

λ_{d} \leq v^{⊤} S v \leq λ_{1} for every v with ∥ v ∥ = 1

The largest possible projected variance is $λ_{1}$ , reached at $v_{1}$ . The smallest is $λ_{d}$ , reached at $v_{d}$ .

TOTAL VARIANCE

tr (S) = j = 1 \sum d S_{j j} = j = 1 \sum d λ_{j}

Total variance is unchanged by rotating to the principal axes. For standardised data it equals $d$ .

PROPORTION OF VARIANCE EXPLAINED

PVE_{j} = \frac{λ _{j}}{\sum _{i = 1}^{d} λ _{i}}, PVE_{1 : k} = \frac{\sum _{j = 1}^{k} λ _{j}}{\sum _{i = 1}^{d} λ _{i}}

Choose the smallest $k$ whose cumulative proportion reaches a stated target, such as 90 percent.

Choosing k

Variance target: the smallest $k$ with $PVE_{1 : k}$ at or above the target. This is how a question usually fixes $k$ .
Scree plot (rule of thumb): plot $λ_{j}$ against $j$ and keep the components before the curve flattens (the elbow).
Eigenvalue above 1 (rule of thumb, correlation matrix only): keep components with $λ_{j} > 1$ , that is, above the average eigenvalue $d / d = 1$ .

PROJECTION (SCORES) ONTO $K$ COMPONENTS

z_{i} = V_{k}^{⊤} (x_{i} - \overset{ˉ}{x}) \in R^{k}, Z = X_{c} V_{k}

$V_{k} = [v_{1} \dots v_{k}]$ has orthonormal columns. The coordinate along $v_{j}$ is $z_{ij} = v_{j}^{⊤} (x_{i} - \overset{ˉ}{x})$ .

RECONSTRUCTION AND ERROR

\hat{x}_{i} = \overset{ˉ}{x} + V_{k} z_{i}, i = 1 \sum n ∥ x_{i} - \hat{x}_{i} ∥^{2} = (n - 1) j = k + 1 \sum d λ_{j}

The $λ_{j}$ are eigenvalues of $S$ with divisor $n - 1$ . With divisor $n$ , the mean squared error $\frac{1}{n} \sum_{i} ∥ x_{i} - \hat{x}_{i} ∥^{2}$ equals the sum of the discarded eigenvalues exactly. Among all $k$ -dimensional affine subspaces (all rank- $k$ linear reconstructions of the centred data), projecting onto the top $k$ components gives the smallest total squared reconstruction error. Maximising the retained variance and minimising the reconstruction error lead to the same components.

EXAM TIP

For a $2 \times 2$ covariance matrix, the eigenvalues satisfy $λ_{1} + λ_{2} = tr (S)$ and $λ_{1} λ_{2} = det (S)$ . This is usually faster than expanding the characteristic polynomial.

EXAM TIP

Adding a constant to any feature changes nothing, because centring removes it. Multiplying every feature by the same $c \neq = 0$ multiplies every eigenvalue by $c^{2}$ and leaves the principal directions and the proportions of variance unchanged. Rescaling only some features generally changes the directions.

PCA through the singular value decomposition

You can get the components without forming $S$ , from the SVD of the centred data matrix. Substitute $X_{c} = U Σ V^{⊤}$ into $X_{c}^{⊤} X_{c}$ to get $X_{c}^{⊤} X_{c} = V Σ^{⊤} Σ V^{⊤}$ . This is an eigendecomposition, and its eigenvectors are the columns of $V$ .

SVD OF THE CENTRED DATA

X_{c} = U Σ V^{⊤}, λ_{j} = \frac{σ _{j}^{2}}{n - 1}

The columns of $V$ (right singular vectors) are the principal directions. The $σ_{j}$ are the singular values in decreasing order. The $λ_{j}$ are eigenvalues of $S$ with divisor $n - 1$ .

SCORES FROM THE SVD

Z = X_{c} V_{k} = U_{k} Σ_{k}

This uses the first $k$ columns of $U$ and $V$ and the top-left $k \times k$ block of $Σ$ . The best rank- $k$ approximation of $X_{c}$ is $U_{k} Σ_{k} V_{k}^{⊤}$ .

GRAM-MATRIX ROUTE (USEFUL WHEN D > N)

X_{c} X_{c}^{⊤} = U Σ Σ^{⊤} U^{⊤}, v_{j} = \frac{X _{c}^{⊤} u _{j}}{σ _{j}} (σ_{j} > 0)

The $n \times n$ matrix $X_{c} X_{c}^{⊤}$ has the same non-zero eigenvalues $σ_{j}^{2}$ as $X_{c}^{⊤} X_{c}$ . When $d > n$ it is the smaller matrix to decompose.

EXAM TIP

Apply the SVD to the centred matrix. The SVD of the uncentred $X$ gives different directions. The columns of $U$ are not the principal directions: they have one entry per sample, and $u_{j} = X_{c} v_{j} / σ_{j}$ is the vector of scores on component $j$ scaled to unit length.

Procedure and worked examples

PCA procedure

Arrange samples in rows. Centre each column, and divide it by its standard deviation if the features differ in scale.
Form $S = \frac{1}{n - 1} X_{c}^{⊤} X_{c}$ , or take the SVD of $X_{c}$ .
Find the eigenvalues in decreasing order and their orthonormal eigenvectors.
Compute the cumulative proportion of variance and choose $k$ .
Project: $Z = X_{c} V_{k}$ . To reconstruct, use $\hat{X} = 1 \overset{ˉ}{x}^{⊤} + Z V_{k}^{⊤}$ .

WORKED EXAMPLE

A centred data set has sample covariance matrix $S = [5222]$ . Find (a) the eigenvalues, (b) the first principal direction (the unit vector with positive components), (c) the proportion of variance explained by the first component, (d) the coordinate along it of the centred point $(3, - 1)^{⊤}$ , and (e) that point's reconstruction from the first component alone, with its squared reconstruction error.

Solution

$tr (S) = 7$ and $det (S) = 10 - 4 = 6$ , so $λ^{2} - 7 λ + 6 = 0$ , which gives $λ_{1} = 6$ and $λ_{2} = 1$ .
For $λ_{1} = 6$ : $(S - 6 I) v = [- 1 2 2 - 4] v = 0$ , so $v_{1} = 2 v_{2}$ . Normalising gives $v_{1} = (2, 1)^{⊤} / 5$ . Check: $S (2, 1)^{⊤} = (12, 6)^{⊤} = 6 (2, 1)^{⊤}$ .
Proportion of variance: $6/ (6 + 1) = 6/7 \approx 0.857$ .
Coordinate of $(3, - 1)^{⊤}$ along $v_{1}$ : $(2 \cdot 3 + 1 \cdot (- 1)) / 5 = 5/ 5 = 5 \approx 2.236$ .
Reconstruction (the data are centred, so $\overset{ˉ}{x} = 0$ ): $5 v_{1} = (2, 1)^{⊤}$ . The error vector is $(3, - 1)^{⊤} - (2, 1)^{⊤} = (1, - 2)^{⊤}$ , so the squared error is $1 + 4 = 5$ .
Check: the second direction is $v_{2} = (1, - 2)^{⊤} / 5$ . The point's coordinate along it is $(3 + 2) / 5 = 5$ , and its square, 5, equals the squared error.

Answer: $λ = 6, 1$ ; $v_{1} = (2, 1)^{⊤} / 5$ ; 6/7 of the variance; coordinate $5 \approx 2.236$ ; reconstruction $(2, 1)^{⊤}$ with squared error 5.

WORKED EXAMPLE

The centred data matrix has the four rows $(1, 0)$ , $(- 1, 0)$ , $(0, 2)$ , $(0, - 2)$ . Find the singular values, the eigenvalues of the sample covariance matrix (divisor $n - 1$ ), and the proportion of variance explained by the first component.

Solution

Each column sums to zero, so the data are centred. $X_{c}^{⊤} X_{c} = [1 + 1 0 0 4 + 4] = [2008]$ .
Its eigenvalues are 8 and 2, so the singular values are $σ_{1} = 8 = 22$ and $σ_{2} = 2$ . The first principal direction is $(0, 1)^{⊤}$ .
With $n = 4$ : $λ_{1} = 8/3$ and $λ_{2} = 2/3$ .
Proportion: $\frac{8/3}{10/3} = 0.8$ .

Answer: $σ = 22, 2$ ; $λ = 8/3, 2/3$ ; the first component explains 80 percent.

What PCA cannot do

Limitation	Why it happens
Misses non-linear structure	Components are linear combinations; data on a curve or a circle needs a non-linear method to be unrolled.
Ignores class labels	High variance need not mean useful for classification: the direction that separates two classes can be a low-variance direction that PCA discards.
Sensitive to scale	Directions generally change when some features are rescaled, so standardise when units differ.
Sensitive to outliers	A few extreme points inflate variance and can swing the first direction.
Components are hard to interpret	Each is usually a mix of all the original features, so the new features have no direct physical meaning.
Uses only second-order statistics	Uncorrelated components are not necessarily independent unless the data are jointly Gaussian.
Sign is arbitrary, repeated eigenvalues are not unique	$v$ and $- v$ are equally valid; if $λ_{j} = λ_{j + 1}$ , any orthonormal basis of that eigenspace works.

EXAM TIP

Keeping all $d$ components loses nothing, because it only rotates the centred data. Information is lost only when $k < d$ . The reconstruction error is proportional to the sum of the discarded eigenvalues. As a fraction of the total squared distance of the points from their mean, it equals $1 - PVE_{1 : k}$ .

EXAM TIP

The rank of $S$ is at most $min (n - 1, d)$ , so with $n$ samples at most $n - 1$ eigenvalues are non-zero. A zero eigenvalue means the data lie exactly in a lower-dimensional flat (an affine subspace).

PREVIOUS-YEAR QUESTIONS

2 questions from the DA papers

Try each question first, then open the answer. We placed each question under a topic ourselves; the exam paper does not label topics.

See these papers in full: GATE DA 2026 · GATE DA 2025

GATE 2026 DA Q.11

GATE 2026 · DAQ.11 · 1 mark · MCQ

For a classification problem, Principal Component Analysis (PCA) has been used to reduce the dimensionality of a feature space from 100 to 10.

Which of the following options is true about the angle $θ$ between the first and the tenth principal components?

(A)
$θ = 0^{\circ}$
(B)
$θ = 9 0^{\circ}$
(C)
$9 0^{\circ} < θ \leq 18 0^{\circ}$
(D)
$0 < θ < 9 0^{\circ}$

Answer and solution

GATE 2026 DA Q.11: answer and solution

See the question in the paper

Tap to open full size

GATE 2026 DA Q.11: answer and solution

GATE 2025 DA Q.60

GATE 2025 · DAQ.60 · 2 marks · NAT

Let $D = {x^{(1)}, \dots, x^{(n)}}$ be a dataset of $n$ observations where each $x^{(i)} \in R^{100}$ . It is given that $\sum_{i = 1}^{n} x^{(i)} = 0$ . The covariance matrix computed from $D$ has eigenvalues $λ_{i} = 10 0^{2 - i}$ , $1 \leq i \leq 100$ . Let $u \in R^{100}$ be the direction of maximum variance with $u^{⊤} u = 1$ .

The value of $\frac{1}{n} \sum_{i = 1}^{n} (u^{⊤} x^{(i)})^{2} =$ ______

(Answer in integer)

Answer and solution

GATE 2025 DA Q.60: answer and solution

See the question in the paper

Tap to open full size

GATE 2025 DA Q.60: answer and solution

PRACTICE

5 questions to test yourself

PRACTICE 11 mark · NAT

A centred data set has sample covariance matrix $S = [3113]$ . What is the variance of the data projected onto the first principal component?

Answer and solution

Answer: $4$ (accepted range $\pm 0.01$ )

$tr (S) = 6$ and $det (S) = 9 - 1 = 8$ , so $λ^{2} - 6 λ + 8 = 0$ .
The eigenvalues are 4 and 2.
The variance along the first principal component is the largest eigenvalue, 4.

PRACTICE 22 marks · MSQ

The data matrix $X_{c} \in R^{n \times d}$ (samples in rows) is centred and has the SVD $X_{c} = U Σ V^{⊤}$ with singular values $σ_{1} \geq σ_{2} \geq \dots$ . The sample covariance uses divisor $n - 1$ . Which of the following are correct?

(A)
The columns of $V$ are the principal directions.
(B)
For each non-zero singular value, $σ_{j}^{2} / (n - 1)$ is an eigenvalue of the sample covariance matrix.
(C)
The matrix of scores on all components is $X_{c} V = U Σ$ .
(D)
The columns of $U$ are the principal directions.

Answer and solution

Answer: (A), (B), (C)

$X_{c}^{⊤} X_{c} = V Σ^{⊤} Σ V^{⊤}$ , so the columns of $V$ are the eigenvectors of the covariance matrix. A is true.
Dividing by $n - 1$ gives the eigenvalues $σ_{j}^{2} / (n - 1)$ . B is true.
$X_{c} V = U Σ V^{⊤} V = U Σ$ . C is true.
The columns of $U$ have one entry per sample; they are normalised score vectors, not directions in feature space. D is false.

PRACTICE 32 marks · NAT

Three points in the plane are $(1, 2)$ , $(3, 4)$ and $(5, 6)$ . After centring, the first principal direction is the unit vector $v_{1}$ with positive components. What is the coordinate of the point $(5, 6)$ along $v_{1}$ (its first principal component score)? Give the answer correct to two decimal places.

Answer and solution

Answer: $2.83$ (accepted range $\pm 0.01$ )

The mean is $(3, 4)$ . The centred points are $(- 2, - 2)$ , $(0, 0)$ and $(2, 2)$ .
$X_{c}^{⊤} X_{c} = [8888]$ , so with divisor $n - 1 = 2$ , $S = [4444]$ . Its eigenvalues are 8 and 0, and the eigenvector for 8 is $v_{1} = (1, 1)^{⊤} / 2$ . All the points lie on the line $y = x$ .
The centred point for $(5, 6)$ is $(2, 2)$ , and its score is $(2 + 2) / 2 = 22 \approx 2.83$ .

PRACTICE 42 marks · MCQ

PCA is applied to four standardised features, so the covariance matrix is the correlation matrix. Its eigenvalues are 2.4, 1.0, 0.4 and 0.2. What is the smallest number of components that retains at least 90 percent of the total variance?

(A)
1
(B)
2
(C)
3
(D)
4

Answer and solution

Answer: (C)

The total variance is $2.4 + 1.0 + 0.4 + 0.2 = 4$ , which equals the number of standardised features.
One component keeps $2.4/4 = 60$ percent.
Two components keep $3.4/4 = 85$ percent, which is below 90 percent.
Three components keep $3.8/4 = 95$ percent, which is at least 90 percent, so $k = 3$ .

PRACTICE 51 mark · MCQ

Which statement about principal component analysis (PCA) on a centred data matrix is correct?

(A)
The first principal component is the direction along which the projected data have the smallest variance.
(B)
The principal directions are eigenvectors of the sample covariance matrix and are mutually orthogonal.
(C)
PCA requires the class labels of the samples to find the components.
(D)
Multiplying one feature by 10 never changes the principal directions of the covariance matrix.

Answer and solution

Answer: (B)

A is false: the first component maximises the projected variance; it does not minimise it.
B is true: the covariance matrix is symmetric, so its eigenvectors can be chosen orthonormal, and these are the principal directions.
C is false: PCA is unsupervised.
D is false: rescaling one feature changes the covariance matrix and generally rotates the directions (they stay put only in special cases, such as uncorrelated features).

OFFICIAL GATE 2027 SYLLABUS WORDING

Copied from the GATE 2027 syllabus; see all 2027 syllabi.

GATE DA

dimensionality reduction, principal component analysis

Preparing in Kolkata? IAE Academy's course for GATE DA.