4

I have recently started looking at inner products. However, I'm still struggling to intuitively understand the definition.

I am aware that in most cases we generalize concepts because they are useful to do certain things. For example, once we realize that properties like convergence and continuity depend on the notion of distance it seems useful to generalize thr concept. Similarly, norms generalizes the idea of magnitude.

In order to generalize something we need to figure out the defining properties of the concept we want to generalize. For example, for the distance between two points:

(i) It is non-negative and only $0$ if the points are the same.

(ii) It is symmetric.

(iii) The shortest path between two points is a line between them.

This is basically the definition of a metric in words rather than using formulas

However, I can't seem to make sense of the definiton of inner products. I know that the algebraic definition of the n-dimensional scalar product

$$ a \cdot b = \sum_{i=1}^n a_1 b_1+\cdots+a_n b_n$$

originally comes from quaternion multiplication and that there is also a geometric definition using the law of cosines, i.e.

$$ a \cdot b = \|a\|\|b\| \cos(\theta)$$ where $\theta<180^\circ$ is the angle between the vectors. This means that it can be used to determine if two vectors are orthogonal which seems to be the reason why one would like to generalize this concept after all (see for example here or here.

The (real) inner product is defined as

(1) $\langle x+y,z\rangle=\langle x,z\rangle+\langle y,z \rangle$

(2) $\langle \alpha x,y\rangle=\alpha\langle x,y\rangle$

(3) $\langle x,y\rangle=\langle y,x\rangle$

(4) $\langle x,x\rangle\geq 0$, $\langle x,x\rangle=0 \iff x=0$

The properties (2) and (3) seem intuitive to me. (3) only says that the relation is symmetric, i.e. if x is orthogonal to y, then y is also orthogonal to x. (2) says that if two vectors are orthogonal to each other, then scaling one of them does not change this fact. However, I can't see the intuition for (1) and (4).

Can someone please shed some light on this?

Edit: By looking at all the answers it became clear that the scalar product should be seen as a scaled version of the scalar projection of $a$ onto $b$.

DerivativesGuy
  • 1,037
  • Regarding (4), note that $\langle x,x\rangle$ is the square of the norm of $x$, which should be $0$ for the $0$ vector and positive otherwise – J. W. Tanner Feb 12 '20 at 18:32
  • Thanks for your comment. Yes, I am aware of this. This is also the case in $\mathbb{R^{n}}$ where we have $x \cdot x = | x |^{2}$. However, I could not see the intuition for this. It kind of just comes from the arithmetic definitions of both. Is there more to it? – DerivativesGuy Feb 12 '20 at 18:37
  • For 4: a vector cannot be perpendicular to itself (unless is has no length) – Andrei Feb 12 '20 at 18:39
  • Yes, that makes sense. But even in $\mathbb{R^{n}}$ I feel like the concept of orthogonality does not make much sense if we do not have two different vectors since there is no angle between a vector and itself. I cannot seem to grasp what it means that the scalar product is the square of the norm if the vectors are the same. – DerivativesGuy Feb 12 '20 at 18:45
  • The inner product $<x,y>$ is the (oriented) length of the orthogonal projection of $x$ on $y$. For $(1)$ draw a parallelogram with sides $x$ and $y$ and project on $z$. – saulspatz Feb 12 '20 at 18:45
  • 1
    @saulspatz use \langle and \rangle for $\langle$ and $\rangle$. – Cameron Williams Feb 12 '20 at 18:46

4 Answers4

5

Here's how you can motivate the coordinate definition of the dot product in $\Bbb R^2$ (for starters) by wanting $x\cdot y = 0$ when $x$ and $y$ are orthogonal. Recall from basic high school geometry that orthogonal lines have slopes that are negative reciprocals of one another (you can get this from basics of similar triangles). So, when $x_1$ and $y_2$ are nonzero (i.e., when the lines are neither horizontal nor vertical), orthogonality is equivalent to $$\frac{x_2}{x_1} = -\frac{y_1}{y_2},$$ which in turn yields $x_1y_1+x_2y_2 = 0$. (And, of course, this formula works fine in the horizontal/vertical case.) Now you get bilinearity and all the rest of the properties, and this shows that $\sum x_iy_i$ is an interesting quantity to study ... :)

Ted Shifrin
  • 115,160
3

Hopefully the following provides some useful perspective:

Think of the inner product as a 'scaled' projection.

Consider $a \cdot b$, and suppose that $\|b\| = 1$. We can write $a= \beta b + w$ where $w \bot b$ and so $a \cdot b = \beta$. In particular (as long as $\|b\|=1$) the quantity $a \cdot b$ gives the projection of $a$ onto the line spanned by $b$.

It is reasonably intuitive that the projection of $a_1+a_2$ should equal to the sum of the projections of the $a_k$, so (1) is reasonable.

From (2) we see that if $a=0$ then $a \cdot a = 0$, and if $a \neq 0$ then we expect $a \cdot a = \|a\| (a \cdot {a \over \|a\|}) = \|a\|^2 >0$.

copper.hat
  • 172,524
  • Thanks for your answer. Could you please elaborate a bit more on what exactly it means that "$a \cdot b$ is the projection of $a$ onto the line spanned by $b$". Thanks very much! – DerivativesGuy Feb 12 '20 at 18:59
  • If $b$ has unit length, then the point $(a \cdot b)b $ is the nearest point to $a$ on the line passing through $b$ and the origin. – copper.hat Feb 12 '20 at 19:01
  • Ah, yes that makes sense. So the scalar product actually gives much more information than just orthogonality. It tells me if I project $a$ onto $b$, then $(a \dot b)b$ is the projected vector. – DerivativesGuy Feb 12 '20 at 19:17
  • Another slightly surprising (and sometimes very useful) thing is that the inner product can be computed in terms of the norm, see https://en.wikipedia.org/wiki/Polarization_identity. – copper.hat Feb 12 '20 at 19:20
1

In the simplest possible terms, the inner product is essentially the "dot product" of an arbitrary vector space. In fact, for a finite-dimensional real vector space (e.g. $\Bbb{R}^n$), the inner product is the dot product. The reason for distinguishing "inner product" as opposed to simply saying "dot product" is that the elementary definition of the dot product...

$$a\cdot b=\sum_n a_nb_n$$

...doesn't always work for an arbitrary vector space. For example, consider the set of all functions $\Bbb{R}\to\Bbb{R}$ with pointwise addition and scalar multiplication defined by $(af)(x)=af(x)$ ($a$ is a scalar, $f$ is a function). This is an infinite-dimensional vector space. Now, suppose that you wanted to find the dot product of two functions $f$ and $g$. Using the formula for the "dot product," you would have...

$$f\cdot g=\sum_{x\in\Bbb{R}}f(x)g(x)$$

Already you see a problem. To calculate the "dot product," you need to take the sum over the set of all real numbers. This is not something that can be done with the basic techniques of analysis, and, in general, is not what we mean when we say "inner product."$^1$ Instead, consider that the "dot product" of two functions is...

$$f\cdot g=\int_{-\infty}^\infty f(x)g(x)\ dx$$

This is an example of an inner product. Of course, not all vector spaces are function spaces, so we need a more general definition that captures the important properties of the "dot product." If you look at the properties shared by the dot product and the "generalised dot product" as defined above, a pattern starts to emerge. If you consider all possible generalisations of the dot product and ask "what does this need?" you end up with the four properties you have listed.

$^1$ There are ways to evaluate such sums, but I've never seen them brought up in the context of vector spaces or discussions of linear algebra.

R. Burton
  • 4,942
  • Thanks for your answer. Yes, I realize that the definition we have in finite-dimensions makes no sense for infinite-dimensional spaces. My problem was merely that I wanted to fully understand why we use these 4 properties and also what they intuitively mean. As you've said we need something general enough to be able to make sense of the concept for any vector space that might come up. @copper.hat has provided some useful intuition regarding these 4 properties. I assume it took mathematicians very long to pin down the exact properties needed to cope with anything that might come up. – DerivativesGuy Feb 12 '20 at 19:26
  • @DerivativesGuy Indeed. If I remember my math history correctly, it wasn't until after the development of Fourier analysis, abstract algebra, and the dawn of quantum mechanics that the notion of "inner product space" reached full maturity. – R. Burton Feb 12 '20 at 19:49
1

To understand the beauty of the inner product (put the intuition aside for now), besides describing the concept as a generalization the OP should know that it is an abstraction, one that 'takes care of' both distance and angles at the same time. I may be biased, but this is a sentence from wikipedia
that 'knocks my socks off',

A Hilbert space $H$ is a real or complex inner product space that is also a complete metric space with respect to the distance function induced by the inner product.

The OP in encouraged to read the history of the how it came about and further developments in that article.

Please indulge me as I pay tribute...

enter image description here

John von Neumann coined the term abstract Hilbert space ...
giving the first complete and axiomatic treatment of them.
-- 1929 --

enter image description here

CopyPasteIt
  • 11,366