Eventually I came up with this java code below, that solves the problem quite fast.
I used the basic Legendre Symbol rule.
$$\left(\frac ap\right)\equiv a^{\frac{p-1}{2}}\pmod p$$
This turns the problem in to calculating $b$ such that $a^c\equiv b(\mod p)$ where $c$ is very large number. I used Billy idea to calculate the mod of power $a$ and than recursively calculate the power of the result $\log c$ times. I used Siddhartha Sharma cleaver approach to implement it.
This is the result. Complexity $O(\log p)$
boolean isRootInQuadraticResidues(BigInteger n, BigInteger p) {
BigInteger tow = BigInteger.valueOf(2);
BigInteger x = n.mod(p);
if (p.equals(tow)) {
return x.mod(tow).equals(BigInteger.ONE);
}
long exponent = p.subtract(BigInteger.ONE).divide(tow).longValue();
return modularExponentiation(x.longValue(), exponent, p.longValue()) == 1;
}
// based on https://math.stackexchange.com/a/453108/101178
long modularExponentiation(long value, long exponent, long mod) {
long result = 1;
while (exponent > 0) {
if ((exponent & 1) == 1) {
value = value % mod;
result = (result * value) % mod;
result = result % mod;
}
exponent = exponent >> 1;
value = value % mod;
value = (value * value) % mod;
value = value % mod;
}
return result;
}