$$AB+AC+BC=N\tag{1}$$
is equivalent to:
$$(2A+B+C)^2-4A^2-(C-B)^2 = 4N\tag{2}$$
hence we may simply look for the integers $q\geq\sqrt{4N}$ that ensure that $q^2-4N$ is the sum of two squares, i.e. is not divided with odd multiplicity by a prime $p\equiv 3\pmod{4}$. For instance, let $N=10$. If we take $q=7$ we have $q^2-4N = 0^2+3^2$, leading to $2A+B+C=7, A=0, C-B=3$ and to $(0,2,5)$ as a trivial solution of $(1)$. If we take $q=8$, we have $q^2-4N=24$ with no associated solutions, since $3\parallel 24$. If we take $q=9$, we have $q^2-4N=4^2+5^2$, leading to $2A+B+C=9, A=2, C-B=5$ and the same trivial solution as before. We may also notice that $(1)$ is equivalent to
$$ (A+C)(B+C) = N+C^2 \tag{3} $$
and design a similar algorithm by factoring $N+C^2$ as the product of two integers in the range $[C,2C]$. That is not possible if $N\geq 3C^2$, hence we may stop searching at $C=\sqrt{\frac{N}{3}}$.
In the previous case, $N=10$, we only have trivial solutions since $11=10+\left\lfloor\frac{10}{3}\right\rfloor$ is a prime.
Moreover, the last algorithm clearly has complexity $\color{red}{O(N^{3/2})}$: we may simply test every integer in the range $[C,2C]$ for being a divisor of $N+C^2$.