2.2: Computing the Estimate
- Page ID
- 24235
The solution, \(\hat{x}\), of Equation 2.1 is characterized by:
\[(y-A \hat{x}) \perp \mathcal{R}\tag{A}\]
All elements in a basis of \(\mathcal{R}(A)\) must be orthogonal to \((y-A \hat{x})\). Equivalently this is true for the set of columns of \(A, [{a_{1}, \dots a_{n}}]\). Thus
\[\begin{aligned}
(y-A \hat{x}) \perp \mathcal{R}(A) & \Leftrightarrow a_{i}^{\prime}(y-A \hat{x})=0 \quad \text { for } i=1, \ldots, n \\
& \Leftrightarrow A^{\prime}(y-A \hat{x})=0 \\
& \Leftrightarrow A^{\prime} A \hat{x}=A^{\prime} y
\end{aligned}\]
This system of m equations in the m unknowns of interest is referred to as the normal equations. We can solve for the unique \(\hat{x}\) iff \(A^{\prime}A\) is invertible. Conditions for this will be derived shortly. In the sequel, we will present the generalization of the above ideas for infinite dimensional vector spaces.