LU & SVD formulas

Line through the origin along $\mathbf{a}\ne 0$; $\mathbf{b}$ and $\mathbf{a}$ are in ${\mathbb{R}}^n$, $P$ is $n\times n$ of rank $1$, and $\mathbf{p}=P\mathbf{b}$.

$A$ is $m\times n$ with linearly independent columns (so $A^{T}A$ is invertible); $P$ is $m\times m$ and projects onto the column space of $A$.

The columns ${\mathbf{q}}_1,\dots ,{\mathbf{q}}_k$ of the $m\times k$ matrix $Q$ are orthonormal ($Q^{T}Q=I_k$); $P$ is $m\times m$ and no inverse is needed.

The normal equations always have a solution; it is unique (and given by the inverse form) only when the columns of $A$ are linearly independent.

Pythagoras applies because the projection and the residual are perpendicular.

For an $m\times k$ matrix $A$ with linearly independent columns this gives $A=QR$: $Q$ is $m\times k$ with orthonormal columns and $R=Q^{T}A$ is $k\times k$ upper triangular with entries $r_{ij}={\mathbf{q}}_i^T{\mathbf{a}}_j$ ($i\le j$) and positive diagonal. The span of the first $j$ vectors is preserved at every step.

Shown for $3\times 3$. Elimination without row exchanges succeeds exactly when the first $n-1$ pivots are non-zero, equivalently when the leading principal minors ${\Delta }_1,\dots ,{\Delta }_{n-1}$ are non-zero; the factorisation with unit lower $L$ is then unique. ($u_{nn}$ may be $0$, which happens exactly when $A$ is singular.)

${\Delta }_k$ is the determinant of the top-left $k\times k$ block of $A$; valid when no row exchanges are needed (${\Delta }_1,\dots ,{\Delta }_{n-1}\ne 0$).

When a zero pivot appears, rows are exchanged; $P$ is an $n\times n$ permutation matrix. A factorisation $PA=LU$ exists for every square $A$, and $\det A=(-1{)}^s{\prod }_i{u}_{ii}$, where $s$ is the number of row exchanges.

Valid when no row exchanges were used; $\det L=1$ because $L$ is unit triangular.

$A$ is symmetric $n\times n$ with ${\Delta }_1,\dots ,{\Delta }_n\ne 0$; then $U=D{L}^T$. $A$ is positive definite exactly when every pivot $u_{kk}>0$ (equivalently every ${\Delta }_k>0$), and then the Cholesky factor $G$ is lower triangular with positive diagonal.

$U$ is $m\times m$ orthogonal (left singular vectors ${\mathbf{u}}_i$), $V$ is $n\times n$ orthogonal (right singular vectors ${\mathbf{v}}_i$), $\Sigma$ is $m\times n$ with the $\min (m,n)$ singular values on its diagonal and zeros elsewhere; $r=\mathrm{rank}A$. Equivalently $A={\sum }_{i=1}^r{\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^T$.

$A^{T}A$ ($n\times n$) and $A{A}^T$ ($m\times m$) are symmetric positive semidefinite and have the same non-zero eigenvalues; all their other eigenvalues are $0$. The columns of $V$ are orthonormal eigenvectors of $A^{T}A$ and the columns of $U$ are orthonormal eigenvectors of $A{A}^T$.

Fixes ${\mathbf{u}}_i$ so that its sign agrees with ${\mathbf{v}}_i$; choosing ${\mathbf{u}}_i$ independently from $A{A}^T$ can give a wrong sign.

$\mathbf{u}\in {\mathbb{R}}^m$ and $\mathbf{v}\in {\mathbb{R}}^n$ are non-zero, so $A$ is $m\times n$ of rank $1$; this follows from $A^{T}A=\Vert \mathbf{u}{\Vert }^2\mathbf{v}{\mathbf{v}}^T$.

$\Vert A{\Vert }_2={\max }_{\Vert \mathbf{x}\Vert =1}\Vert A\mathbf{x}\Vert$ is the spectral norm; $\Vert A{\Vert }_F$ is the Frobenius norm (square root of the sum of squared entries).

$A$ is $n\times n$; $A$ is invertible exactly when ${\sigma }_n>0$, and ${\kappa }_2$ is the 2-norm condition number.

Keeps the $k$ largest singular values, $k\le r$; $A_k$ has rank $k$.

$B$ ranges over $m\times n$ matrices; the minimum is attained by $B=A_k$ in both norms; take ${\sigma }_{k+1}=0$ if $k\ge r$.