Eigen decomposition, Singular Value Decomposition (SVD) and Principal Component Analysis

 1. Eigen decomposition for a square matrix:

   Av = λv:
     an eigenvector v is a direction that A only scales
     (by eigenvalue λ) without rotation.

   To find eigenvalues λ, solve det(A - λI) = 0.

   Then express A = VΛV⁻¹:
     eigen decomposition rewrites A in terms of its eigenvectors V
     and eigenvalues Λ.

2. Singular Value Decomposition (SVD) for any matrix:

   Eigen decomposition of XᵀX:
     eigenvectors = right singular vectors V,
     eigenvalues  = squared singular values σ².

   Eigen decomposition of XXᵀ:
     eigenvectors = left singular vectors U,
     eigenvalues  = squared singular values σ².

   Full factorization:
     X = UΣVᵀ:
       U (m×m) = left singular vectors,
       V (n×n) = right singular vectors,
       Σ (m×n) = diagonal matrix with singular values σi ≥ 0.

3. Principal Component Analysis (PCA):

   Application: to find the directions of maximum variance
   (the principal components) in the data.

   Mathematics: Covariance C = XᵀX, then perform eigen decomposition of C.

   With X = UΣVᵀ:
     - Right singular vectors V = principal components,
     - Squared singular values σ² = eigenvalues of the covariance C,
     - Projections of samples onto components = UΣ.