Chapter 5

5.1 Orthogonality

5.1.1 Orthogonal Subspaces

Two vectors $v,w\in {\mathbb{R}}^n$ are called orthogonal if $v^{\top }w={\sum }_{i=1}^n{v}_i{w}_i=0$. Two subspaces $V$ and $W$ are orthogonal if for all $v\in V$ and $w\in W$, the vectors $v$ and $w$ are orthogonal.

5.1.2 Subspaces Orthogonal if Basis is Orthogonal

Let $v_1,\dots ,v_k$ be a basis of subspace $V$. Let $w_1,\dots ,w_l$ be a basis of subspace $W$. $V$ and $W$ are orthogonal if and only if $v_i$ and $w_j$ are orthogonal for all $i\in \{1,\dots ,k\}$ and $j\in \{1,\dots ,l\}$.

As all vectors are pairwise linearly independent, their set is also linearly independent.

5.1.3 Basis of Subspace together form basis of ${\mathbb{R}}^n$.

Let $V$ and $W$ be orthogonal subspaces of ${\mathbb{R}}^n$. Let $v_1,\dots ,v_k$ be a basis of subspace $V$. Let $w_1,\dots ,w_l$ be a basis of subspace $W$. The set of vectors $\{v_1,\dots ,v_k,w_1,\dots ,w_l\}$ are linearly independent.

We can use this Lemma to get the basis of the subspace $V+W$, which is then indeed also a subspace of ${\mathbb{R}}^n$. This sum of orthogonal vectors is called the Minkovsky-Sum!

5.1.4 Properties of orthogonal subspaces

Let $V$ and $W$ be orthogonal subspaces. Then $V\cap W=\{0\}$ (as all subspaces contain the $0$ vector).

If $\dim (V)=k$ and $\dim (W)=l$, then $\dim (V+W)=k+l\le n$.

5.1.5 Orthogonal Complement

Let $V$ be a subspace of ${\mathbb{R}}^n$. We define the orthogonal complement of $V$ as $V^{\perp }=\{w\in {\mathbb{R}}^n|w^{\top }v=0\text{for all }v\in V\}$

This $V^{\perp }$ is also a subspace of ${\mathbb{R}}^n$. Therefore, the concept of orthogonal subspaces allows us to decompose the space ${\mathbb{R}}^n$ into a subspace and it’s complement (these being the nullspace of $A$ and the row space of $A$).

5.1.6 Decomposition of ${\mathbb{R}}^{m\times n}$

Let $A\in {\mathbb{R}}^{m\times n}$ be a matrix. $N(A)=C(A^{\top }{)}^{\perp }=R(A)$

As we know that if $r=\dim (R(A))$ then $n-r=\dim (N(a))$. Thus they add up to the whole space!

5.1.7 Orthogonal subspace properties

Let $V,W$ be orthogonal subspaces of ${\mathbb{R}}^n$. Then the following statements are equivalent.

• $W=V^{\perp }$
• $\dim (V)+\dim (W)=n$
• Every $u\in {\mathbb{R}}^n$ can be written as $u=v+w$ with unique vectors $v\in V$, $w\in W$

5.1.8 Orthogonality cancels out

Let $V$ be a subspace of ${\mathbb{R}}^n$. Then $V=(V^{\perp }{)}^{\perp }$

This allows us to rewrite Theorem 5.1.6:

5.1.8 Four fundamental Subspaces

Let $A\in {\mathbb{R}}^{m\times n}$. $N(A)=C(A^{\top }{)}^{\perp }\text{and}C(A^{\top })=N(A{)}^{\perp }$

We can visualise the four fundamental subspaces in the following way:

Consequences: Recall that the complete solution to $Ax=b$ (“all $x$‘s st $Ax=b$“) $\text{Sol}(A,b)=\{x\in {\mathbb{R}}^n\mid x=x_p+x_n,x_n\in N(A)\}=x_p+N(A)=\text{"one }{x}_p\text{ + any }{x}_n\text{"}$ where $x_p$ is a particular solution (assuming the system is feasible).

You can show that $\text{Sol}(A,b)$ is also equal to $x_r+N(A)=\text{"}{x}_r\text{ + any }{x}_n\text{"}$, where $x_r\in C(A^T)=R(A)$ and $x_n\in N(A)$ and where $x_r\in C(A^T)$ is unique (i.e. such a particular solution $x_p=x_r\in R(A)$ always exists and is unique) (See Sabatino Notes Week 9 for Proof.)

5.1.10 $A$ and $A^{\top }A$

Let $A\in {\mathbb{R}}^{m\times n}$. Then $N(A)=N(A^{\top }A)$ and $C(A^{\top })=C(A^{\top }A)$.

This holds because:

• if $x\in N(A)$ then $Ax=0⟹A^{\top }Ax=A\cdot 0⟹A^{\top }Ax=0$.
• if $x\in N(A^{\top }A)$ then $A^{\top }Ax=0$, which means $0=x^{\top }0=x^{\top }{A}^{\top }Ax=(Ax{)}^{\top }(Ax)=||Ax|{|}^2⟹Ax=0$

5.2 Projections

If a system $Ax=b$ has no solutions (because $A$ is not invertible, i.e. $b\notin C(A)$) can we find a $p\in C(A)$ such that the error is minimal - basically the next best solution?

This usually happens when we have more equations than variables (i.e. each is being “overdetermined”) $m\ge n$.

5.2.1 Projection of a vector onto a subspace

The projection of a vector $b\in {\mathbb{R}}^m$ onto a subspace $S$ (of ${\mathbb{R}}^m$) is the point in $S$ that is closest to $b$. In other words ${\text{proj}}_S(b)={\text{argmin}}_{p\in S}||b-p||$

Where $b=p+e⟹b-p=e$, with $e$ the error.

The one-dimensional case

We can clearly see that the error vector in 2d is orthogonal to the projection $e^{\top }a=(b-p{)}^{\top }a$ by geometric intuition.

5.2.2 Projection in 2d formula

Let $a\in {\mathbb{R}}^m\backslash \{0\}$. The projection of $b\in {\mathbb{R}}^m$ on $S=\{\lambda a|\lambda \in \mathbb{R}\}=C(a)$ is given by ${\text{proj}}_S(b)=\frac{a{a}^{\top }}{a^{\top }a}b$ This minimiser is unique.

And indeed if we substitute we find that the error vector is indeed orthogonal to $a$.

Note that if we want to find the projection of a vector $b=\lambda a$, then the projection simply returns $\lambda a$, as $b$ is already on the line (subspace).

General Case

5.2.3 Projection is well defined

The projection of a vector $b\in {\mathbb{R}}^m$ to the subspace $S=C(A)$ is well defined. It can be written as ${\text{proj}}_S(b)=A\hat{x}$ where $\hat{x}$ satisfies the normal equations $A^{\top }A\hat{x}=A^{\top }b$

INFDEF

If $A^{T}A$ is invertible, we have a unique solution $\hat{x}$ that satisfies the equation: $p=A\hat{x}=A(A^{\top }A{)}^{-1}{A}^{\top }b$

5.2.4 $A^{\top }A$ invertible $\iff$ $A$ has indep. columns

$A^{\top }A$ is invertible if and only if $A$ has linearly independent columns.

5.2.5 Projection Matrix

Let $S$ be a subspace in ${\mathbb{R}}^m$ and $A$ a matrix whose columns are a basis of $S$. The projection of $b\in {\mathbb{R}}^m$ to $S$ is given by ${\text{proj}}_S(b)=Pb$ where $P=A(A^{\top }A{)}^{-1}{A}^{\top }$ is the projection matrix.

Note the condition for the columns to be a basis - this forces them to be independent, which means $A^{\top }A$ invertible by Lemma 5.2.4.

IMPORTANT It may look like we can simplify the expression for the projection matrix $P$. This is not the case as $(A\top A{)}^{-1}=A^{-1}(A^{\top }{)}^{-1}$ is only the case if $A$ itself is invertible. But if $A$ is invertible, it spans ${\mathbb{R}}^m$ anyways and any projection is simply the point itself. This is beautifully reflected in the fact that if we simplify $P=A{A}^{-1}(A^{\top }{)}^{-1}{A}^{\top }$ then we simply get $P=I$.

5.2.6

• If $b\in {\mathbb{R}}^m$ then ${\text{proj}}_S({\text{proj}}_S(b))={\text{proj}}_S(b)$ by definition. This requires us to have that $PPb=Pb\iff P^2=P$.
• Let $S^{\perp }$ be the orthogonal complement of $S$ and $P$ the projection matrix onto $S$.
  • Then $I-P$ is the projection matrix that maps $b\in {\mathbb{R}}^m$ to ${\text{proj}}_{S^{\perp }}(b)$.
  • Proof Idea: Since $b=e+{\text{proj}}_S(b)=e+Pb$ with $e\in S^{\perp }$ Thus $(I-P)b=b-Pb=e={\text{proj}}_{S^{\perp }}(b)$
• Note that - as it should be - we have that $(I-P{)}^2=I-2P+P^2=I-P-P+P=I-P$