In the paper "A Monte Carlo method for factorization", Pollard introduces a method for factoring composite numbers that is better than trial division. The main idea is to generate random numbers and test if the difference between any pair has a non-$1$ divisor in common with the number to be factored, which exploits the birthday paradox to find factors faster than trial division.
In his paper, Pollard uses the formula: $x_{i+1} \equiv x_i^2 - 1 \bmod n$, to generate random numbers, where $n$ is the number that we want to factor and $x_0 = 2$. He notes that any polynomial of degree $\geq 2$ can be used.
It intuitively makes sense to me that linear functions can have additional properties that cause the generated sequence to not be random enough. However, I see no clear reason why the algorithm would only work for polynomials. Using different functions, for example with bit manipulations and storing intermediate results modulo $2^{64}$ could potentially make the algorithm faster on real computers.
I have tried implementing the algorithm using built-in random number generators of my programming language (the splitmix PRNG in particular) and noticed that the algorithm performs much worse. I don't think it is likely that this is due to poor statistical properties of splitmix, because it is mature and it passes many statistical tests. It seems more likely to me that there is a deeper mathematical reason that makes polynomials more suitable for this algorithm, but I cannot pinpoint why that is the case.
So, my question is: why does the Pollard rho algorithm for factorizing composite numbers work much better with PRNG's based on polynomials?