PCA formulas

Dimensionality reduction and principal component analysis. 13 formulas for GATE DA Machine Learning, each with when to use it. Each is explained in the topic notes.

All 13 PCA formulas

SAMPLE MEAN AND CENTRED DATA

\overset{ˉ}{x} = \frac{1}{n} i = 1 \sum n x_{i}, x_{i}^{c} = x_{i} - \overset{ˉ}{x}

Centre each feature (column) to mean zero. $X_{c}$ denotes the centred matrix, whose rows are $(x_{i}^{c})^{⊤}$ .

SAMPLE COVARIANCE MATRIX

S = \frac{1}{n - 1} X_{c}^{⊤} X_{c} \in R^{d \times d}

Symmetric and positive semi-definite. Diagonal entries are feature variances; off-diagonal entries are covariances. Some texts divide by $n$ instead. The eigenvectors and the proportions of variance do not change; every eigenvalue is scaled by the same factor.

PRINCIPAL DIRECTIONS

S v_{j} = λ_{j} v_{j}, λ_{1} \geq λ_{2} \geq \dots \geq λ_{d} \geq 0, v_{i}^{⊤} v_{j} = 0 (i \neq = j)

The eigenvectors of the symmetric matrix $S$ are chosen orthonormal. $v_{j}$ is the $j$ -th principal direction (component). A direction is fixed only up to sign.

SPECTRAL DECOMPOSITION AND COVARIANCE OF SCORES

S = V Λ V^{⊤} = j = 1 \sum d λ_{j} v_{j} v_{j}^{⊤}, \frac{1}{n - 1} Z^{⊤} Z = Λ_{k}

$V = [v_{1} \dots v_{d}]$ is orthogonal and $Λ = diag (λ_{1}, \dots, λ_{d})$ . The scores $Z = X_{c} V_{k}$ (defined below) have the diagonal covariance $Λ_{k} = diag (λ_{1}, \dots, λ_{k})$ .

VARIANCE ALONG A PRINCIPAL DIRECTION

Var (v_{j}^{⊤} x^{c}) = v_{j}^{⊤} S v_{j} = λ_{j}

Projections onto different components are uncorrelated: $v_{i}^{⊤} S v_{j} = 0$ for $i \neq = j$ .

VARIANCE ALONG ANY UNIT DIRECTION

λ_{d} \leq v^{⊤} S v \leq λ_{1} for every v with ∥ v ∥ = 1

The largest possible projected variance is $λ_{1}$ , reached at $v_{1}$ . The smallest is $λ_{d}$ , reached at $v_{d}$ .

TOTAL VARIANCE

tr (S) = j = 1 \sum d S_{j j} = j = 1 \sum d λ_{j}

Total variance is unchanged by rotating to the principal axes. For standardised data it equals $d$ .

PROPORTION OF VARIANCE EXPLAINED

PVE_{j} = \frac{λ _{j}}{\sum _{i = 1}^{d} λ _{i}}, PVE_{1 : k} = \frac{\sum _{j = 1}^{k} λ _{j}}{\sum _{i = 1}^{d} λ _{i}}

Choose the smallest $k$ whose cumulative proportion reaches a stated target, such as 90 percent.

PROJECTION (SCORES) ONTO $K$ COMPONENTS

z_{i} = V_{k}^{⊤} (x_{i} - \overset{ˉ}{x}) \in R^{k}, Z = X_{c} V_{k}

$V_{k} = [v_{1} \dots v_{k}]$ has orthonormal columns. The coordinate along $v_{j}$ is $z_{ij} = v_{j}^{⊤} (x_{i} - \overset{ˉ}{x})$ .

RECONSTRUCTION AND ERROR

\hat{x}_{i} = \overset{ˉ}{x} + V_{k} z_{i}, i = 1 \sum n ∥ x_{i} - \hat{x}_{i} ∥^{2} = (n - 1) j = k + 1 \sum d λ_{j}

The $λ_{j}$ are eigenvalues of $S$ with divisor $n - 1$ . With divisor $n$ , the mean squared error $\frac{1}{n} \sum_{i} ∥ x_{i} - \hat{x}_{i} ∥^{2}$ equals the sum of the discarded eigenvalues exactly. Among all $k$ -dimensional affine subspaces (all rank- $k$ linear reconstructions of the centred data), projecting onto the top $k$ components gives the smallest total squared reconstruction error. Maximising the retained variance and minimising the reconstruction error lead to the same components.

SVD OF THE CENTRED DATA

X_{c} = U Σ V^{⊤}, λ_{j} = \frac{σ _{j}^{2}}{n - 1}

The columns of $V$ (right singular vectors) are the principal directions. The $σ_{j}$ are the singular values in decreasing order. The $λ_{j}$ are eigenvalues of $S$ with divisor $n - 1$ .

SCORES FROM THE SVD

Z = X_{c} V_{k} = U_{k} Σ_{k}

This uses the first $k$ columns of $U$ and $V$ and the top-left $k \times k$ block of $Σ$ . The best rank- $k$ approximation of $X_{c}$ is $U_{k} Σ_{k} V_{k}^{⊤}$ .

GRAM-MATRIX ROUTE (USEFUL WHEN D > N)

X_{c} X_{c}^{⊤} = U Σ Σ^{⊤} U^{⊤}, v_{j} = \frac{X _{c}^{⊤} u _{j}}{σ _{j}} (σ_{j} > 0)

The $n \times n$ matrix $X_{c} X_{c}^{⊤}$ has the same non-zero eigenvalues $σ_{j}^{2}$ as $X_{c}^{⊤} X_{c}$ . When $d > n$ it is the smaller matrix to decompose.

Where this was asked: 2025 (1), 2026 (1). See the 2 previous-year questions on PCA. This shows what was asked, not what will be asked.

KEEP GOING

PCA: the notes and 2 solved previous-year questions