A concave quadrilateral cannot be circumscribed by an ellipse nor by a parabola.

Question

It's intuitive that a concave quadrilateral cannot be circumscribed by an ellipse nor by a parabola.

I can prove it by analytic geometry:

Without loss of generality, let $L_1\equiv m_1x -y +r_1=0$, $L_2\equiv m_2x -y +r_2=0$, $L_3\equiv m_3x -y +r_3=0$, $L_4\equiv m_4x -y +r_4=0$ be the equations of lines $AB$, $BC$, $CD$, $DA$ of a concave quadrilateral ABCD.

Since ABCD is a concave quadrilateral, by virtue of the theorem stated and proved in The concave quadrilateral and the slopes of its sides, we have

$$(m_1-m_2)(m_2-m_3)(m_3-m_4)(m_4-m_1)<0$$

Besides that, all the conics which circumscribe the concave quadrilateral ABCD can be given by the equation $kL_1L_3+L_2L_4=0$ (except the degenerate conic consisting of the pair of concurrent lines $AB$ and $CD$). This degenerate conic is a degenerate hyperbole because a concave quadrilateral cannot have parallel opposite sides.

Therefore all the conics circumscribing the quadrilateral ABCD (except the mentioned degenerate hyperbole) are given by the equation $$k(m_1x -y +r_1)(m_3x -y +r_3)+(m_2x -y +r_2)(m_4x -y +r_4)=0,$$ $$(m_1m_3k +m_2m_4)x^2-((m_1+m_3)k+(m_2+m_4))xy+(k+1)y^2+...=0$$

The type of this circumscribing conic is given by

$$\delta=(km_1m_3+m_2m_4)(k+1)-\frac 14(k(m_1+m_3)+(m_2+m_4))^2,$$

This circumscribing conic is an ellipse if $\delta>0$, an hyperbole if $\delta<0$, a parabola if $\delta=0$.

Developing $\delta$ we get

$$4\delta= 4m_1m_3k^2+4m_2m_4k+4m_1m_3k+4m_2m_4-((m_1+m_3)^2k^2+2(m_1+m_3)(m_2+m_4)k+(m_2+m_4)^2),$$ $$4\delta=-(m_1-m_3)^2k^2+2[2m_2m_4+2m_1m_3-(m_1+m_3)(m_2+m_4)]k-(m_2-m_4)^2,$$

$$4\delta=-(m_1-m_3)^2k^2+2[(m_1-m_3)(m_2-m_4)+2(m_1-m_4)(m_3-m_2)]k-(m_2-m_4)^2$$

Now let $\psi$ be the discriminant of this second degree equation in k: $$\psi=4[(m_1-m_3)(m_2-m_4)+2(m_1-m_4)(m_3-m_2)]^2-4(m_1-m_3)^2(m_2-m_4)^2,$$ $$\frac 14 \psi=[(m_1-m_3)(m_2-m_4)+2(m_1-m_4)(m_3-m_2)]^2-(m_1-m_3)^2(m_2-m_4)^2,$$ $$\frac 14 \psi=4(m_1-m_3)(m_2-m_4)(m_1-m_4)(m_3-m_2)+4(m_1-m_4)^2(m_3-m_2)^2,$$ $$\frac 1{16} \psi=(m_1-m_3)(m_2-m_4)(m_1-m_4)(m_3-m_2)+(m_1-m_4)^2(m_3-m_2)^2,$$ $$\frac 1{16} \psi=(m_1-m_4)(m_3-m_2)[(m_1-m_3)(m_2-m_4)+(m_1-m_4)(m_3-m_2)],$$ $$\frac 1{16} \psi=(m_1-m_4)(m_3-m_2)(m_1-m_2)(m_3-m_4),$$

$$\frac 1{16} \psi=(m_1-m_2)(m_2-m_3)(m_3-m_4)(m_4-m_1)$$

Hence this discriminant $\psi$ is negative and since $-(m_1-m_3)^2<0$, $\delta$ is always negative. Consequently a concave quadrilateral can only be circumscribed by hyperboles, therefore it cannot be circumscribed by an ellipse nor by a parabola,

QED.

Is there another formal proof of it?

Answer 1

I have no idea about the parabola, I'll only cover the case of ellipse.

Since "concaveness" is preserved under linear transform, we can use a linear transform to map the ellipse to the unit circle centered at origin. The question becomes "can we inscribe a concave quadrilateral into that unit circle?".

Let $ABCD$ be a concave simple quadrilateral. Since it is concave, one of its vertices, say $D$, lies in the interior of the triangle $ABC$. This means we can find $\alpha, \beta, \gamma > 0$ such that $$D = \alpha A + \beta B + \gamma C\quad\text{ and }\quad\alpha+\beta+\gamma + 1$$

Let $\lambda = \alpha+\beta$ and $\mu = \frac{\alpha}{\lambda}$, we have $\lambda,\mu \in (0,1)$ and $(\alpha, \beta, \gamma) = (\lambda\mu,\lambda(1-\mu),1-\lambda)$

Let $\varphi(P)$ be the squared distance of point $P$ from origin. When $A,B,C$ lies on the unit circle, we have $\varphi(A) = \varphi(B) = \varphi(C) = 1$. In order to inscribe $ABCD$ into the unit circle, we also need $\varphi(D) = 1$.

As a function of $P$, $\varphi(P)$ is strictly convex over the Euclidean plane. This implies

$$\begin{align} \varphi(D) &= \varphi(\alpha A + \beta B + \gamma C) = \varphi(\lambda(\mu A + (1-\mu)B) + (1-\lambda) C)\\ &< \lambda \varphi(\mu A + (1-\mu)B) + (1-\lambda) \varphi(C)\\ &< \lambda( \mu \varphi(A) + (1-\mu)\varphi(B)) + (1-\lambda) \varphi(C)\\ &= \lambda( \mu + (1-\mu) ) + 1-\lambda\\ &= 1 \end{align} $$ As a result, $D$ falls inside the interior of unit disk and cannot lie on the unit circle.

---

Update

I find an argument that works for both parabola and ellipse. The key is the concept extreme point of a convex set. There are several easy to verify facts.

1. If $D$ lies in the interior of triangle $ABC$, then $D$ is not an extreme point for the triangle (which is a convex set).

2. If $P \in X \subset Y$ for convex sets $X$ and $Y$, then $P$ is an extreme point for $Y$ implies $P$ is an extreme point for $X$.

3. The region bounded by an ellipse is convex and the ellipse is the set of extreme points for this region.

Let's say we have an ellipse $\mathcal{E}$ circumscribe a concave quadrilateral $ABCD$. Since $ABCD$ is concave, one of its vertices, say $D$ lies in the interior of triangle $ABC$.

• By $(1)$, $D$ is not an extreme point for triangle $ABC$.
• By $(2)$, $D$ is not an extreme point for the region bounded by $\mathcal{E}$ (since this region contains triangle $ABC$).
• By $(3)$, $D$ doesn't lie on $\mathcal{E}$.

This contradict with the assumption that $\mathcal{E}$ circumscribe $ABCD$.

For parabola, the argument is similar. A parabola divides the plane into two components. One component of it is convex while the other concave. It is easy to see if a parabola $\mathcal{P}$ circumscribe an quadrilateral $ABCD$, then the interior of the edges $AB$, $BC$, $CD$, $DA$ and body of quadrilateral lies inside the convex component.

We can replace $(3)$ by an alternate fact.

4. The parabola is the set of extreme points for the corresponding convex component.

Essentially same argument like before tell us if $ABCD$ is concave, then the point $D$ (the one lies inside the triangle formed by other points) cannot lies on $\mathcal{P}$. A contradiction again!