41. Orthogonal Basis,¹ Gram-schmidt Process, Orthonormal Basis¹

Dated: 19-06-2025

Example

Let \(\mathbb W = \text{Span}(\{x_1, x_2\})\) where

\[ x_1 = \begin{bmatrix} 3\\ 6\\ 0 \end{bmatrix} \text{ and } x_2 = \begin{bmatrix} 1\\ 2\\ 2 \end{bmatrix} \]

Find the orthogonal basis¹ \(\{v_1, v_2\}\) for \(\mathbb W\).

Solution

Let \(\vec v_1 = \vec x_1\) and \(P\) be projection¹ of \(\vec x_2\) on \(\vec x_1\). The component which is orthogonal¹ to to \(\vec x_1\) is \(\vec v_2 = \vec x_2 - P\)

\[ x_2 - \frac{x_2 \cdot v_1}{v_1 \cdot v_1} v_1 = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} - \frac{15}{45} \begin{bmatrix} 3 \\ 6 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 2 \end{bmatrix} \]

Gram Schmidt Process

Example

The following vectors² \(\{\vec x_1, \vec x_2, \vec x_3\}\) are linearly independent.³

\[ x_1 = \begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \end{bmatrix}, x_2 = \begin{bmatrix} 0 \\ 1 \\ 1 \\ 1 \end{bmatrix}, x_3 = \begin{bmatrix} 0 \\ 0 \\ 1 \\ 1 \end{bmatrix} \]

Construct an orthogonal basis¹ for \(W\) by Gram-Schmidt Process.

Solution

To construct orthogonal basis¹ we have to perform the following steps.

Step 1

Let \(v_1 = x_1\)

Step 2

Let \(v_2 = x_2 - \frac{x_2 \cdot v_1}{v_1 \cdot v_1} v_1\)

\[ v_2 = x_1 = \begin{bmatrix} 0 \\ 1 \\ 1 \\ 1 \end{bmatrix} - \frac{3}{4} \begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \end{bmatrix} = \begin{bmatrix} \frac {-3} 4 \\ \frac 1 4 \\ \frac 1 4 \\ \frac 1 4 \end{bmatrix} = \begin{bmatrix} -3 \\ 1 \\ 1 \\ 1 \end{bmatrix} \]

Step 3

\[ v_3 = x_3 - \left[ \frac{x_3 \cdot v_1}{v_1 \cdot v_1} v_1 + \frac{x_3 \cdot v_2}{v_2 \cdot v_2} v_2 \right] = \begin{bmatrix} 0 \\ 0 \\ 1 \\ 1 \end{bmatrix} - \left[ \frac{2}{4} \begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \end{bmatrix} + \frac{2}{12} \begin{bmatrix} -3 \\ 1 \\ 1 \\ 1 \end{bmatrix} \right] = \begin{bmatrix} 0 \\ \frac 2 3 \\ \frac 2 3 \\ \frac 2 3 \end{bmatrix} \]

\[ v_3 = \begin{bmatrix} 0 \\ 0 \\ 1 \\ 1 \end{bmatrix} - \begin{bmatrix} 0 \\ \frac 2 3 \\ \frac 2 3 \\ \frac 2 3 \end{bmatrix} = \begin{bmatrix} 0 \\ \frac {-2} 3 \\ \frac 1 3 \\ \frac 1 3 \end{bmatrix} \]

Thus \(\{v_1, v_2, v_3\}\) is an orthogonal basis¹ for \(\mathbb W\).

Theorem

Give a basis \(\{x_1, x_2, \ldots, x_p\}\) for subspace⁴ \(\mathbb W\) for \(\mathbb R^n\).

\[ v_1 = x_1 \]

\[ \begin{array}{ccc} v_2 &= &x_2 - \frac{x_2 \cdot v_1}{v_1 \cdot v_1} v_1 \\ v_3 &= &x_3 - \frac{x_3 \cdot v_1}{v_1 \cdot v_1} v_1 - \frac{x_3 \cdot v_2}{v_2 \cdot v_2} v_2\\ &\vdots \\ v_p &= &x_p - \frac{x_p \cdot v_1}{v_1 \cdot v_1} v_1 - \frac{x_p \cdot v_2}{v_2 \cdot v_2} v_2 - \dots - \frac{x_p \cdot v_{p-1}}{v_{p-1} \cdot v_{p-1}} v_{p-1} \end{array} \]

Then \(\{v_1, \ldots, v_p\}\) is the orthogonal basis¹ for \(\mathbb W\).

\(\text{Span}(\{v_1, \ldots, v_p\}) = \text{Span}(\{x_1, \ldots, x_k\}) \quad 1 \le k \le p\)

Theorem

If \(A\) is an \(m \times n\) matrix⁵ with linearly independent³ columns then \(A = QR\)
Where \(Q_{m \times n}\) has columns which form an orthogonal basis¹ for \(\text{Col } A\) and \(R_{n \times n}\) is an upper triangular invertible matrix⁶⁷ with positive entries on its diagonal.

Example

Find \(QR\) factorization of

\[ A = \begin{bmatrix} 1 & 2 & 5 \\ -1 & 1 & -4 \\ -1 & 4 & -3 \\ 1 & -4 & 7 \\ 1 & 2 & 1 \end{bmatrix} \]

Solution

First find the orthonormal basis¹ by applying Gram Schmidt process on the columns of \(A\).

\[ Q = \begin{bmatrix} \frac 1 {\sqrt{5}} & \frac 1 2 & \frac 1 2 \\ \frac {-1} {\sqrt{5}} & 0 & 0 \\ \frac {-1} {\sqrt{5}} & \frac 1 2 & \frac 1 2 \\ \frac 1 {\sqrt{5}} & \frac{-1} 2 & \frac 1 2 \\ \frac 1 {\sqrt{5}} & \frac 1 2 & \frac{-1} 2 \end{bmatrix} \]

\[ R = Q^T A = \begin{bmatrix} \sqrt{5} & -\sqrt{5} & 4\sqrt{5} \\ 0 & 6 & -2 \\ 0 & 0 & 4 \end{bmatrix} \]

References

Read more about notations and symbols.

---

1. Read more about orthogonal and orthonormal basis. ↩↩↩↩↩↩↩↩↩↩↩

2. Read more about vectors. ↩

3. Read more about linear dependency. ↩↩

4. Read more about vector spaces. ↩

5. Read more about matrices. ↩

6. Read more about upper triangular matrix. ↩

7. Read more about invertible matrices. ↩