Foundations of linear algebra · Section 2 of 5
Inner products & Cauchy–Schwarz
Where length and angle come from; the whole inequality follows from “a projection is never longer”.
about 7 min
Start from a problem
The eight axioms of a vector space mention only addition and scaling. There is no "length" and no "angle". Yet the most basic geometric questions are exactly these: how long is this vector? what is the angle between two vectors? are they perpendicular?
We need to add a structure to a vector space from which length and angle can be defined, and which works just as well in spaces of polynomials, functions or random variables that do not look like arrows.
In the plane, the projection of onto the direction of has length . Whatever is, the projection is never longer than itself. This sounds too obvious to mention, yet it is the entire source of this section's inequality.
Drag and below. The blue arrow is the projection of onto the line of ; the dashed red segment is what is left over. When does the bar reach the bar, and what does the gap between them have to do with the red segment?
- u
- v
- projection tv
- remainder u − tv
- unit “circle” ‖x‖ = 1
What did you observe?Play first, then answer. Read the explanation only after you have thought it through.
- 1.Drag u once around the origin. When is the |⟨u,v⟩| bar as long as ‖u‖‖v‖? How are u and v related then?
- 2.Pull u away from the line through v. How is the “slack” related to the length of the dashed red remainder u − tv?
- 3.Switch to the weighted inner product 2u₁v₁ + u₂v₂ and press “Orthogonal when weighted”. u and v do not look perpendicular, yet their inner product is 0. What has the unit circle become? Does the inequality still hold?
Given a "product" with a few natural properties, we can define
Definitions
An inner product on a real vector space is a map
- symmetry:
; - linearity:
; - positive definiteness: , with equality iff .
A vector space with an inner product is an inner product space.
- with
(the standard inner product). - Continuous functions on with
. - Random variables with finite variance,
.
All three axioms check out in each case. Even has more than one:
Theorem and proof
In any inner product space,
with equality iff are linearly dependent.
If both sides are . Let and set
Substituting merges the last two terms:
Multiply by . Equality holds iff (positive definiteness), i.e. ; together with the case , that is linear dependence.
The proof used exactly two things: lengths are non-negative (axiom 3) and subtract the projection (axioms 1 and 2 to expand). So it holds, word for word, in every inner product space.
Multiplying the proof's first line by gives an exact formula for how much room the inequality leaves:
This is the "slack" in the experiment: the gap between the two bars is controlled by the length of the dashed red remainder, and it closes exactly when the remainder vanishes.
.
Only with the triangle inequality does deserve the name "length"; only with Cauchy–Schwarz is
The equality case of Cauchy–Schwarz is "linearly dependent", not "equal". gives equality; so does (then , but the squares agree). And the clause "equality iff " in axiom 3 cannot be dropped: without it could vanish for and the projection coefficient would be undefined.
Applications
Substituting the three examples:
The third says the correlation coefficient lies in . All three are the same proof.
The point of a subspace closest to is the orthogonal projection of onto . The heart of the proof is again "what remains after subtracting the projection is orthogonal to ". That is the entire geometry of linear regression.
This section shows algebra's typical move: do not ask what an inner product is, ask what it satisfies, and let one proof cover every case. The next section, Linear maps, treats "transformation" with the same attitude.
Exercises
0 of 2 done
✓ All exercises in this section are done.
Next section: Linear maps →
Discussion
Found a mistake, have a question or a suggestion? You can also write to 620026600@qq.com
No comments yet. Share your understanding, ask a question, or point out an error on this page.
Log in to join the discussion