Singular Value Decomposition

PCA can be determined by using the singular value decomposition (SVD) [5]

$$\mathbf{X}= \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T \, ,$$ (3)

where $\mathbf{U}$ is a $d\times d$ orthogonal matrix, $\mathbf{V}$ is an $n\times n$ orthogonal matrix and $\mathbf{\Sigma}$ is a $d\times n$ pseudodiagonal matrix (diagonal if $d = n$) with the singular values on the main diagonal [5]. The PCA solution is obtained by selecting the $c$ largest singular values from $\mathbf{\Sigma}$, by forming $\mathbf{A}$ from the corresponding $c$ columns of $\mathbf{U}$, and $\mathbf{S}$ from the corresponding $c$ rows of $\mathbf{\Sigma}\mathbf{V}^T$.

Note that PCA can equivalently be defined using the eigendecomposition of the $d\times d$ covariance matrix $\mathbf{C}$ of the column vectors of the data matrix $\mathbf{X}$:

$$\mathbf{C}= \frac{1}{n} \mathbf{X}\mathbf{X}^T = \mathbf{U}\mathbf{D}\mathbf{U}^T \, ,$$ (4)

Here, the diagonal matrix $\mathbf{D}$ contains the eigenvalues of $\mathbf{C}$, and the columns of the matrix $\mathbf{U}$ contain the unit-length eigenvectors of $\mathbf{C}$ in the same order [6,4,2,5]. Again, the columns of $\mathbf{U}$ corresponding to the largest eigenvalues are taken as $\mathbf{A}$, and $\mathbf{S}$ is computed as $\mathbf{A}^T \mathbf{X}$. This approach can be more efficient for cases where $d\ll n$, since it avoids the $n\times n$ matrix.