1.2: Vector Spaces

Review the definition of a vector space: vectors, field of scalars, vector addition (which must be associative and commutative), scalar multiplication (with its own associativity and distributivity properties), the existence of a zero vector 0 such that x + 0 = x for every vector x, and the normalization conditions 0x = 0, 1x = x. Use the definition to understand that the first four examples below are vector spaces, while the fifth and sixth are not:

• \(\mathbf{R}^{n} \text { and } \mathbf{C}^{n}\)
• Real continuous functions f(t) on the real line (\(\forall t\)), with obvious definitions of vector addition (add the functions pointwise, f(t) + g(t)) and scalar multiplication (scale the function by a constant, af (t)).
• The set of m x n matrices.
• The set of solutions y(t) of the LTI ode .\[y^{(1)}(t)+3 y(t)=0.\]
• The set of points \(\left[\begin{array}{lll}x_{1} & x_{2} & x_{3}\end{array}\right]\) in satisfying \(x_{1}^{2}+x_{2}^{2}+x_{3}^{2}=1\), i.e. "vectors" from the origin to the unit sphere.
• The set of solutions y(t) of the LTI ode \(y^{(1)}(t)+3 y(t)=\sin t\).

A subspace of a vector space is a subset of vectors that itself forms a vector space. To verify that a set is a subspace, all we need to check is that the subset is closed under vector addition and under scalar multiplication; try proving this. Give examples of subspaces of the vector space examples above.

• Show that the range of any real n x m matrix and the nullspace of any real m x n matrix are subspaces of \(\mathbf{R}^{n}\).
• Show that the set of all linear combinations of a given set of vectors forms a subspace (called the subspace generated by these vectors, also called their linear span)
• Show that the intersection of two subspaces of a vector space is itself a subspace.
• Show that the union of two subspaces is in general not a subspace. Also determine under what condition the union of subspaces will be a subspace.
• Show that the (Minkowski or) direct sum of subspaces, which by definition comprises vectors that can be written as the sum of vectors drawn from each of the subspaces, is a subspace.

Get in the habit of working up small (in \(\mathbf{R}^{2}\) or \(\mathbf{R}^{3}\), for instance) concrete examples for yourself, as you tackle problems such as the above. This will help you develop a feel for what is being stated- perhaps suggesting a strategy for a proof of a claim, or suggesting a counterexample to disprove a claim.

Review what it means for a set of vectors to be (linearly) dependent or (linearly) independent. A space is n-dimensional if every set of more than n vectors is dependent, but there is some set of n vectors that is independent; any such set of n independent vectors is referred to as a basis for the space.

• Show that any vector in an n-dimensional space can be written as a unique linear combination of the vectors in a basis set; we therefore say that any basis set spans the space.
• Show that a basis for a subspace can always be augmented to form a basis for the entire space.

If a space has a set of n independent vectors for every non negative n, then the space is called infinite dimensional.

• Show that the set of functions f(t) = \(t^{n-1}\), n = 1,2,3, ... forms a basis for an infinite dimensional space. (One route to proving this uses a key property of Vandermonde matrices, which you may have encountered somewhere.)

Norms

The "lengths" of vectors are measured by introducing the idea of a norm. A norm for a vector space \(\mathcal{V}\) over the field of real numbers R or complex numbers C is defined to be a function that maps vectors x to nonnegative real numbers \(\|x\|\), and that satisfies the following properties:

1. Positivity: \(\|x\|>0\) for \(x \neq 0\)
2. Homogeneity: \(\|a x\|=|a| \| x \mid\) for scalar \(a\)
3. Triangle inequality: \(\|x+y\| \leq\|x\|+\|y\|\) with \(\forall x, y \in \mathcal{V}\)

• Verify that the usual Euclidean norm on \(\mathbf{R}^{n}\) or \(\mathbf{C}^{n}\) (namely \(\sqrt{x^{\prime} x}\) with ' denoting the complex conjugate of the transpose) satisfies these conditions.
• A complex matrix Q is termed Hermitian if Q' = Q; if Q is real, then this condition simply states that Q is symmetric. Verify that \(x^{\prime} Q x\) is always real, if Q is Hermitian. A matrix is termed positive definite if \(x^{\prime} Q x\) is real and positive for \(x \neq 0\). Verify that \(\sqrt{x^{\prime} Q x}\) constitutes a norm if Q is Hermitian and positive definite.
• Verify that in \(\mathbf{R}^{n}\) both \(\|x\|_{1}=\sum_{1}^{n}\left|x_{i}\right|\) and \(\|x\|_{\infty}=\max _{i}\left|x_{i}\right|\) constitute norms. These are referred to as the 1-norm and \(\infty\)-norm respectively, while the examples of norms mentioned earlier are all instances of (weighted or unweighted) 2-norms. Describe the sets of vectors that have unit norm in each of these cases.
• The space of continuous fucntions on the interval [0, 1] clearly forms a vector space. One possible norm defined on this space is the \(\infty\)-norm defined as:

\[\|f\|_{\infty}=\sup _{t \in[0,1]} \mid f(t)\]

This measures the peak value of the function in the interval [0, 1]. Another norm is the 2-norm defined as:

\[\|f\|_{2}=\left(\int_{0}^{1}|f(t)|^{2} d t\right)^{\frac{1}{2}}\]

Verify that these measures satisfy the three properties of the norm.

Inner Product

The vector spaces that are most useful in practice are those on which one can define a notion of inner product. An inner product is a function of two vectors, usually denoted by < x, y > where x and y are vectors, with the following properties:

1. Symmetry: \(\left\langle x, y>=\langle y, x\rangle^{\prime}\right.\)
2. Linearity: \(\langle x, a y+b z>=a<x, y>+b<x, z>\) for all scalars \(a\) and \(b\)
3. Positivity: \(<x, x>\) is positive for \(x \neq 0\)

• Verify that \(\sqrt{\langle x, x\rangle}\) defines a norm.
• Verify that \(x^{\prime} Q y\) constitutes an inner product if \(Q\) is Hermitian and positive definite. The case of \(Q = I\) corresponds to the usual Euclidean inner product.
• Verify that \[\int_{0}^{1} x(t) y(t) d t\] defines an inner product on the space of continuous functions. In this case, the norm generated from this inner product is the same as the 2-norm defined earlier.
• Cauchy-Schwartz Inequality Verify that for any x and y in an inner product space \(|<x, y>| \leq\|x\| \| y \mid\) with equality if and only if \(x=\alpha y\) for some scalar \(\alpha\). (Hint: Expand \(<x+\alpha y, x+\alpha y>\)).

Two vectors x, y are said to be orthogonal if \(\langle x, y>=0\); two sets of vectors \(\mathcal{X}\) and \(\mathcal{Y}\) are called orthogonal if every vector in one is orthogonal to every vector in the other. The orthogonal complement of a set of vectors \(\mathcal{X}\) is the set of vectors orthogonal to \(\mathcal{X}\), and is denoted by \(\mathcal{X}^{\perp}\).

• Show that the orthogonal complement of any set is a subspace.