LeoMath

Foundations of linear algebra · Section 2 of 5

Inner products & Cauchy–Schwarz

Where length and angle come from; the whole inequality follows from “a projection is never longer”.

about 7 min

Start from a problem

Problem

The eight axioms of a vector space mention only addition and scaling. There is no "length" and no "angle". Yet the most basic geometric questions are exactly these: how long is this vector? what is the angle between two vectors? are they perpendicular?

We need to add a structure to a vector space from which length and angle can be defined, and which works just as well in spaces of polynomials, functions or random variables that do not look like arrows.

Observe

In the plane, the projection of u onto the direction of v has length ‖u‖|cos⁡θ|. Whatever θ is, the projection is never longer than u itself. This sounds too obvious to mention, yet it is the entire source of this section's inequality.

Drag u and v below. The blue arrow is the projection of u onto the line of v; the dashed red segment is what is left over. When does the |⟨u,v⟩| bar reach the ‖u‖‖v‖ bar, and what does the gap between them have to do with the red segment?

Interactive experimentProjection and the Cauchy–Schwarz slack
  • u
  • v
  • projection tv
  • remainder u − tv
  • unit “circle” ‖x‖ = 1
|⟨u,v⟩|5.5
‖u‖‖v‖8.51
⟨u,v⟩ 5.5cos θ 0.646projection coefficient t 0.55slack 42.25
⟨u,v⟩2≤‖u‖2‖v‖2‖u‖2‖v‖2−⟨u,v⟩2=‖v‖2‖u−tv‖2
inner product
Arrow keys move

What did you observe?Play first, then answer. Read the explanation only after you have thought it through.

  1. 1.Drag u once around the origin. When is the |⟨u,v⟩| bar as long as ‖u‖‖v‖? How are u and v related then?
  2. 2.Pull u away from the line through v. How is the “slack” related to the length of the dashed red remainder u − tv?
  3. 3.Switch to the weighted inner product 2u₁v₁ + u₂v₂ and press “Orthogonal when weighted”. u and v do not look perpendicular, yet their inner product is 0. What has the unit circle become? Does the inequality still hold?
Conjecture

Given a "product" ⟨u,v⟩ with a few natural properties, we can define ‖u‖=⟨u,u⟩ and cos⁡θ=⟨u,v⟩‖u‖‖v‖. For the latter to make sense we must prove |⟨u,v⟩|≤‖u‖‖v‖, and "a projection never gets longer" should be the proof.

Definitions

Definition 2.1Inner product

An inner product on a real vector space V is a map ⟨⋅,⋅⟩:V×V→ℝ such that for all u,v,w∈V and c∈ℝ:

  1. symmetry: ⟨u,v⟩=⟨v,u⟩;
  2. linearity: ⟨cu+w,v⟩=c⟨u,v⟩+⟨w,v⟩;
  3. positive definiteness: ⟨v,v⟩≥0, with equality iff v=0.

A vector space with an inner product is an inner product space.

Definition 2.2Length, angle, orthogonality

‖v‖=⟨v,v⟩ is the length (norm) of v. If ⟨u,v⟩=0, u and v are orthogonal.

Example 2.1Three inner product spaces
  • ℝn with ⟨a,b⟩=∑aibi (the standard inner product).
  • Continuous functions on [0,1] with ⟨f,g⟩=∫01f(x)g(x)dx.
  • Random variables with finite variance, ⟨X,Y⟩=𝔼[XY].

All three axioms check out in each case. Even ℝ2 has more than one: ⟨u,v⟩=2u1v1+u2v2 is an inner product too (you can switch to it in the experiment), with its own lengths and right angles. In the third, ‖X−𝔼X‖2 is the variance.

Theorem and proof

Theorem 2.1Cauchy–Schwarz inequality

In any inner product space,

⟨u,v⟩2≤⟨u,u⟩⟨v,v⟩,

with equality iff u,v are linearly dependent.

Proof

If v=0 both sides are 0. Let v≠0 and set t=⟨u,v⟩⟨v,v⟩, the coefficient of the projection of u onto v. "What remains after subtracting the projection has non-negative length":

0≤⟨u−tv, u−tv⟩=⟨u,u⟩−2t⟨u,v⟩+t2⟨v,v⟩.

Substituting t merges the last two terms:

0≤⟨u,u⟩−⟨u,v⟩2⟨v,v⟩.

Multiply by ⟨v,v⟩>0. Equality holds iff u−tv=0 (positive definiteness), i.e. u=tv; together with the case v=0, that is linear dependence.

The proof used exactly two things: lengths are non-negative (axiom 3) and subtract the projection (axioms 1 and 2 to expand). So it holds, word for word, in every inner product space.

Multiplying the proof's first line by ⟨v,v⟩ gives an exact formula for how much room the inequality leaves:

‖u‖2‖v‖2−⟨u,v⟩2=‖v‖2‖u−tv‖2.

This is the "slack" in the experiment: the gap between the two bars is controlled by the length of the dashed red remainder, and it closes exactly when the remainder vanishes.

Corollary 2.2Triangle inequality

‖u+v‖≤‖u‖+‖v‖.

Proof

‖u+v‖2=‖u‖2+2⟨u,v⟩+‖v‖2≤‖u‖2+2‖u‖‖v‖+‖v‖2=(‖u‖+‖v‖)2.

Only with the triangle inequality does ‖⋅‖ deserve the name "length"; only with Cauchy–Schwarz is ⟨u,v⟩‖u‖‖v‖∈[−1,1], so that cos⁡θ is defined. Angle grows out of the inner product, not the other way round.

Common mistake

The equality case of Cauchy–Schwarz is "linearly dependent", not "equal". u=3v gives equality; so does u=−v (then ⟨u,v⟩<0, but the squares agree). And the clause "equality iff v=0" in axiom 3 cannot be dropped: without it ⟨v,v⟩ could vanish for v≠0 and the projection coefficient t would be undefined.

Applications

ApplicationThree faces of one inequality

Substituting the three examples:

(∫fg)2≤∫f2∫g2,|Cov⁡(X,Y)|≤Var⁡XVar⁡Y.

The third says the correlation coefficient lies in [−1,1]. All three are the same proof.

ApplicationLeast squares

The point of a subspace W closest to u is the orthogonal projection of u onto W. The heart of the proof is again "what remains after subtracting the projection is orthogonal to W". That is the entire geometry of linear regression.

Remark

This section shows algebra's typical move: do not ask what an inner product is, ask what it satisfies, and let one proof cover every case. The next section, Linear maps, treats "transformation" with the same attitude.

Exercises

0 of 2 done

1
Find the cosine of the angle between u=(1,2,2) and v=(2,−1,2) (three decimals).

Decimals or fractions; anything within 0.0006 of the answer counts as correct.

2
For reals a1,…,an, Cauchy–Schwarz gives (a1+⋯+an)2≤C(a12+⋯+an2). The smallest such C is
Answer to exercise 2

Discussion

Found a mistake, have a question or a suggestion? You can also write to 620026600@qq.com

No comments yet. Share your understanding, ask a question, or point out an error on this page.

Log in to join the discussion