I'm not aware of any cipher which uses square root as a primitive operation, and I suspect that none exist. Cipher designers tend to prefer a large number of simple operations to a lesser number of complex ones. The most complex operation used in practice is multiplication, and even that is not extremely common. The most complex mathematical operations I know of used in a cipher are exponentiation and logarithms. From a quote by James Massey on the use of complex functions:
I have many times used the discrete exponential or the discrete logarithm as nonlinear cryptographic functions and they have never let me down.
There is no reason why a square root operation would need to return a floating point value (for example, in 8-bit integer arithmetic, $\sqrt50=7$, though it's likely that taking the fractional part of the square root would be more beneficial for causing diffusion and promoting nonlinearity), so it is possible to use, technically. It's perfectly possible to return an integer and still achieve diffusion. I have a few guesses as to why no cipher uses square roots as a cryptographic mixing operation:
It's difficult to write a fast constant time implementation, which is necessary to avoid side-channels.
The nonlinear properties it provides are not well-researched compared to other operations.
It is a complex and slow function, and cryptographers often prefer many simple and fast functions.
Unless done in a finite field, it is impossible to reverse the operation (which is often useful).
If you loosen your requirements, then the Rabin cryptosystem may fit the bill. It's an asymmetric cipher and is very similar to RSA, but with the public exponent fixed to $2$. This means that decryption involves computing modular square roots. It's very possible that there is other asymmetric cryptography which makes use of square root operations, but I can't think of any off the top of my head.
If you can expand your requirements to any $n^\text{th}$-root operations, then the broken KN-Cipher from 1995 may be relevant. Its round function is based on the cube root operation in $\operatorname{GF}(2^{33})$. Computing cube roots is also a simpler and faster process than computing square roots.