Suppose a plane quadrilateral ABCD without sides parallel to y-axis, and let $m_1, m_2, m_3, m_4$ be the slopes of the equations of sides AB, BC, CD, DA (the cartesian axes being orthogonal or oblique). Having made these definitions, we may state the following theorem:
ABCD is a concave quadrilateral iff $$(m_1-m_2)(m_2-m_3)(m_3-m_4)(m_4-m_1)<0$$
This theorem (which I found out on my own) can be proved by a reasoning which takes in consideration the position of the vertices of the quadrilateral in relation to the diagonals, but I would appreciate if someone could show me a different way of proving it.
Proof
Here we will prove a stronger theorem: Let $L_1=p_1x+q_1y+r_1=0$, $L_2=p_2x+q_2y+r_2=0$, $L_3=p_3x+q_3y+r_3=0$, $L_4=p_4x+q_4y+r_4=0$ be cartesian equations of sides AB, BC, CD, DA of a plane quadrilateral ABCD. Then we may state the following theorem:
ABCD is a concave quadrilateral iff $$\begin{vmatrix}p_1 & q_1\\p_2 & q_2\\\end{vmatrix}\begin{vmatrix}p_2 & q_2\\p_3 & q_3\\\end{vmatrix}\begin{vmatrix}p_3 & q_3\\p_4 & q_4\\\end{vmatrix}\begin{vmatrix}p_4 & q_4\\p_1 & q_1\\\end{vmatrix}<0$$
Let $A=(x_1,y_1)$, $B=(x_2,y_2)$, $C=(x_3,y_3)$, $D=(x_4,y_4)$ be the cartesian coordinates of the vertices of the quadrilateral.
Equations of the diagonal lines:
$$AC(x,y)\equiv\begin{vmatrix}x & y & 1\\x_1 & y_1 & 1\\x_3 & y_3 & 1\\\end{vmatrix}=0$$
$$BD(x,y)\equiv\begin{vmatrix}x & y & 1\\x_2 & y_2 & 1\\x_4 & y_4 & 1\\\end{vmatrix}=0$$
Equation of side AB:
$$AB(x,y)\equiv\begin{vmatrix}x & y & 1\\x_1 & y_1 & 1\\x_2 & y_2 & 1\\\end{vmatrix}=0,$$ $$AB(x,y)\equiv (y_1-y_2)x+(x_2-x_1)y+(x_1y_2-x_2y_1)=0$$
As this equation and the equation $L_1=p_1x+q_1y+r_1=0$ given above represent the same line, $\exists$ $k_1\neq 0$ such that $$(y_1-y_2)=k_1p_1, (x_2-x_1)=k_1q_1.$$
Likewise $\exists$ $k_2$, $k_3$, $k_4$, all different from zero, such that $$(y_2-y_3)=k_2p_2, (x_3-x_2)=k_2q_2.$$ $$(y_3-y_4)=k_3p_3, (x_4-x_3)=k_3q_3.$$ $$(y_4-y_1)=k_4p_4, (x_1-x_4)=k_4q_4.$$
That being done, we note that each diagonal line of a quadrilateral splits the plane in two half-planes, and how the respective remaining vertices locate relating to the diagonal line it's a key to classify the quadrilateral:
Lemma I. A plane quadrilateral is convex iff each of its diagonal lines splits the respective remaining vertices in distinct half-planes.
Lemma II. A plane quadrilateral is crossed iff each of its diagonal lines doesn't split the respective remaining vertices, both staying in the same half-plane.
Lemma III. A plane quadrilateral is concave iff one diagonal line splits the remaining vertices in distinct half-planes, and the other diagonal line doesn't split the remaining vertices.
Translating these purely geometric lemmas into the algebraic language of analytic geometry we get
I.ABCD is convex iff $AC(x_2,y_2).AC(x_4,y_4)<0$ and $BD(x_1,y_1).BD(x_3,y_3)<0$ .
II. ABCD is crossed iff $AC(x_2,y_2).AC(x_4,y_4)>0$ and $BD(x_1,y_1).BD(x_3,y_3)>0$.
III. ABCD is concave iff $AC(x_2,y_2).AC(x_4,y_4).BD(x_1,y_1).BD(x_3,y_3)<0$
Therefore ABCD is concave
$$\Leftrightarrow \begin{vmatrix}x_2 & y_2 & 1\\x_1 & y_1 & 1\\x_3 & y_3 & 1\\\end{vmatrix}\begin{vmatrix}x_4 & y_4 & 1\\x_1 & y_1 & 1\\x_3 & y_3 & 1\\\end{vmatrix}\begin{vmatrix}x_1 & y_1 & 1\\x_2 & y_2 & 1\\x_4 & y_4 & 1\\\end{vmatrix}\begin{vmatrix}x_3 & y_3 & 1\\x_2 & y_2 & 1\\x_4 & y_4 & 1\\\end{vmatrix}<0$$
$$\Leftrightarrow \begin{vmatrix}x_1-x_2 & y_1-y_2\\x_3-x_2 & y_3-y_2\\\end{vmatrix}\begin{vmatrix}x_1-x_4 & y_1-y_4\\x_3-x_4 & y_3-y_4\\\end{vmatrix}\begin{vmatrix}x_2-x_1 & y_2-y_1\\x_4-x_1 & y_4-y_1\\\end{vmatrix}\begin{vmatrix}x_2-x_3 & y_2-y_3\\x_4-x_3 & y_4-y_3\\\end{vmatrix}<0$$
$$\Leftrightarrow \begin{vmatrix}-k_1q_1 & k_1p_1 \\k_2q_2 & -k_2p_2\\\end{vmatrix}\begin{vmatrix}k_4q_4 & -k_4p_4 \\-k_3q_3 & k_3p_3 \\\end{vmatrix}\begin{vmatrix}k_1q_1 & -k_1p_1 \\-k_4q_4 & k_4p_4\\\end{vmatrix}\begin{vmatrix}-k_2q_2 & k_2p_2\\k_3q_3 & -k_3p_3\\\end{vmatrix}<0$$
$$\Leftrightarrow (k_1k_2k_3k_4)^2\begin{vmatrix}p_2 & q_2 \\p_1 & q_1\\\end{vmatrix}\begin{vmatrix}p_3 & q_3 \\p_4 & q_4\\\end{vmatrix}\begin{vmatrix}p_4 & q_4 \\p_1 & q_1\\\end{vmatrix}\begin{vmatrix}p_3 & q_3\\p_2 & q_2\\\end{vmatrix}<0$$
$$\Leftrightarrow \begin{vmatrix}p_1 & q_1 \\p_2 & q_2\\\end{vmatrix}\begin{vmatrix}p_2 & q_2 \\p_3 & q_3\\\end{vmatrix}\begin{vmatrix}p_3 & q_3 \\p_4 & q_4\\\end{vmatrix}\begin{vmatrix}p_4 & q_4\\p_1 & q_1\\\end{vmatrix}<0$$
QED.
Thus, if no side of ABCD is parallel to y-axis, ABCD is concave
$$\Leftrightarrow \begin{vmatrix}m_1 & -1\\m_2 & -1\\\end{vmatrix}\begin{vmatrix}m_2 & -1\\m_3 & -1\\\end{vmatrix}\begin{vmatrix}m_3 & -1\\m_4 & -1\\\end{vmatrix}\begin{vmatrix}m_4 & -1\\m_1 & -1\\\end{vmatrix}<0$$
$$\Leftrightarrow (m_1-m_2)(m_2-m_3)(m_3-m_4)(m_4-m_1)<0$$
QED.
Has anyone ever seen this theorem in a book, paper or in the internet before?
Is anyone acquainted with a different proof?