1.3: The Projection Theorem

Consider the following minimization problem:

\[\min _{m \in M}\|y-m\|\]

where the norm is defined through an inner product. The projection theorem (suggested by the figure below), states that the optimal solution \(\hat{m}\) is characterized as follows:

\[(y-\hat{m}) \perp M\]

To verify this theorem, assume the converse. Then there exists an \(m_{0},\left\|m_{0}\right\|\) = 1, such that \(<y-\hat{m}, m_{0}>=\delta \neq 0\). We now argue that \(\left(\hat{m}+\delta m_{0}\right) \in M\) achieves a smaller value to the above minimization problem. In particular,

\[\begin{aligned}
\left\|y-\hat{m}-\delta m_{0}\right\|^{2} &=\|y-\hat{m}\|^{2}-<y-\hat{m}, \delta m_{0}>-<\delta m_{0}, y-\hat{m}>+|\delta|^{2}\left\|m_{0}\right\|^{2} \\
&=\|y-\hat{m}\|^{2}-|\delta|^{2}-|\delta|^{2}+|\delta|^{2} \\
&=\|y-\hat{m}\|^{2}-|\delta|^{2}
\end{aligned}\]

This contradicts the optimality of \(\hat{m}).

• Given a subspace \(\mathcal{S}\), show that any vector x can be uniquely written as \(x=x_{\mathcal{S}}+x_{\mathcal{S}^{\perp}}\), where \(x_{\mathcal{S}} \in \mathcal{S}\) and \(x_{\mathcal{S} \perp} \in \mathcal{S}^{\perp}\).