Summary of least-squares solution methods

We have identified three methods to solve the least-squares problem.

Method 1: normal equation for tall skinny matrix $A$:

$$x=(A^{T}A{)}^{-1}{A}^{T}b$$

Method 2: recommended method using QR:

$$\begin{array}{c}A=QR\\ x=R^{-1}{Q}^{T}b\end{array}$$

Method 3: for rank deficient $A$, we use the SVD:

$$\begin{array}{c}A=U\Sigma V^T\\ x=V{\Sigma }^{-1}{U}^{T}b\end{array}$$