Let $V$ be an $K$-linear space with inner product, $S \subset V$ a linear subspace and $V/S$ the quotient space. I want to prove that the quadratic form $\psi: V/S \to K$ with $\psi([v])=min\{\|v'\|^2: [v]=[v']\}$ defines an inner product in the quotient.
Should I use the polarization formula $\varphi(v,w)=\frac{1}{4}(\psi(v+w)-\psi(v-w))$ to define the bilinear form, and then use that to prove the properties of the inner product? I'm really lost in that question. Thanks.
As @user8675309 pointed out in their comment, the question features a mistake of category: after all, an inner product on the vector space $V/S$ is defined to be a map \begin{align} \varphi:V/S\times V/S&\to\mathbb K\\ (x,y)&\mapsto\varphi(x,y) \end{align} with certain properties. In particular it needs two arguments, while $\psi$ as defined in the question only takes one argument as input. Hence the question is not "Is $\psi$ an inner product?" -- because that would mix two different types of objects together -- but rather "Can $\psi$ somehow generate an inner product on $V/S$, and if so, how?". And for that second question your idea already goes into the right direction.
In terms of category, $\psi$ is closer to a norm on $V/S$ (one input, non-negative output) so
to me seems like the best course of action.
One readily verifies that given any $v\in V$ $$ \psi([v])=\min_{s\in S}\|v+s\|^2=\min_{s\in S}\langle v+s,v+s\rangle $$ and thus $\psi(\alpha x)=|\alpha|^2\psi(x)$ for all $\alpha\in\mathbb K$, $x\in V/S$. But a norm needs to be homogeneous, that is, $\|\alpha x\|=|\alpha|\|x\|$ which suggests taking the square root of $\psi$. Hence the object of interest becomes \begin{align} \sqrt\psi:V/S&\to[0,\infty)\\ x&\mapsto\min_{y\in x}\|y\|\,. \end{align} Let us quickly show that $\sqrt\psi$ is indeed a norm: it is homogeneous (by construction), positive definite (if $\sqrt\psi(x)=0$, then $0\in x$ by definiteness of $\|\cdot\|$, so $x=[0]$), and satisfies the triangle inequality: for all $v,w\in V$ \begin{align} \sqrt\psi([v]+[w])=\sqrt\psi([v+w])&=\min_{s\in S}\|v+w+s\|\\ &\leq \min_{s\in S}\Big(\Big\|v+\frac{s}2\Big\|+\Big\|w+\frac{s}2\Big\|\Big)\\ &\leq \Big(\min_{s\in S}\Big\|v+\frac{s}2\Big\|\Big)+\Big(\min_{s\in S}\Big\|w+\frac{s}2\Big\|\Big)\\ &=\big(\min_{\tilde s\in S}\|v+\tilde s\|\big)+\big(\min_{\tilde s\in S}\|w+\tilde s\|\big)=\sqrt\psi([v])+\sqrt\psi([w])\,. \end{align} Now because $\sqrt\psi$ is a norm on $V/S$, all that is left to show is that it satisfies the parallelogram law, i.e. for all $x,y\in V/S$ $$ 2\psi(x)+2\psi(y)=\psi(x+y)+\psi(x-y)\tag{1} $$ Then it would generate an inner product on $V/S$ via \begin{align} \varphi:V/S\times V/S&\to\mathbb K\\ (x,y)&\mapsto\frac{ (\sqrt\psi(x+y))^2+(\sqrt\psi(x-y))^2 }4=\frac{ \psi(x+y)+\psi(x-y)}4\tag{2} \end{align} if $\mathbb K=\mathbb R$ (and analogously if $\mathbb K=\mathbb C$).
In order to show that (1) holds, we have to "explicitly" calculate $\psi([v])$. For this we need the following auxiliary result which is another formulation of the Hilbert projection theorem:
Lemma. Given $v\in V$ and any orthonormal basis $(z_i)_{i=1}^m$ of the orthogonal complement $S^\perp$ of $S$ in $V$, one has $$ \psi([v])=\sum_{i=1}^m|\langle z_i,v\rangle|^2\,. $$ Proof: By the projection theorem the minimizer $s_v\in S$ of $\min_{s\in S}\|v-s\|$ is the unique element in $S$ such that $v-s\in S^\perp$. Hence $s_v=\pi_S(v)$ is the orthogonal projection of $v$ onto $S$, and using the Pythagorean theorem we compute $$ \psi([v])=\|v-\pi_S(v)\|^2=\|\pi_{S^\perp}(v)\|^2=\Big\|\sum_{i=1}^m\langle z_i,v\rangle z_i\Big\|^2=\sum_{i=1}^m|\langle z_i,v\rangle|^2\,.\tag*{$\square$} $$
With this we can verify (1) by means of a direct computation: given any orthonormal basis $(z_i)_{i=1}^m$ of $S^\perp$ one (in abuse of notation) finds \begin{align} \psi([v+w])+\psi([v-w])&=\sum_{i=1}^m|\langle z_i,v\pm w\rangle|^2\\ &=\sum_{i=1}^m\langle z_i,v\pm w\rangle\langle v\pm w,z_i\rangle\\ &= \sum_{i=1}^m\big(\langle z_i,v\rangle\langle v,z_i\rangle\underbrace{\pm \langle z_i,v\rangle\langle w,z_i\rangle}_{\text{cancel out}}\pm \underbrace{\langle z_i,w\rangle\langle v,z_i\rangle}_{\text{cancel out}}+ \langle z_i,w\rangle\langle w,z_i\rangle\big)\\ &=2\sum_{i=1}^m|\langle z_i,v\rangle|^2+2\sum_{i=1}^m|\langle z_i,w\rangle|^2=2\psi([v])+2\psi([w])\,. \end{align} And with this we are done -- as explained above, because $\sqrt\psi$ is a norm which satisfies the parallelogram law it generates an inner product on $V/S$ via (2).
