This is an interesting question. I will provide a method to simplify your algorithm, but not necessarily a formula just yet (I'm sure that what I'm about to show you will lead to a formula...probably).
Let's begin with a point in the center. That is, we are not starting at the top corner of a semi-infinite plane, we are instead assuming the plane is infinite. The point in the center is assigned the number 1, and we call it the $i = 1$ point. We surround this square with a border of squares. This border has 8 such squares. We repeat the process and get 16 squares. In general, each "border" has $2(2i-1) + 2(2i-3) = 8i - 8 = 8(i-1)$ squares.
We now want to form a sum of the first $N$ such squares:
$$S_N = \bigg(\sum_{i=2}^{N}8(i-1)\bigg) + 1$$
Now, you want to simplify your life, so let's simplify that sum. We would end up with:
$$S_N = 8\bigg(\sum_{i=2}^{N}(i-1)\bigg) + 1$$
$$S_N = 8\frac{N(N+1)}{2}-1 - (8N - 8) + 1$$ (I'll leave you to simplify) and check my algebra. I'm 99% sure it's accurate.
Now you want to unpack this. This involves several steps. Say you have the number $M$. You need to find the largest $N$ such that $S_N \le M \lt S_{N+1}$. Other than a strict search, I'm afraid I don't know how to do that, sorry.
Once you know the $N$, then you need to compute the quantity $M - S_N$. This quantity tells you how many squares to "walk" from some starting square on border $N+1$ to where you want to be. Since you know where the border squares start, and you know how far you've walked, you know where on the border you are and therefore what the coordinates are.
The method needs some cleanup, but should do it. Good luck doing that in N-D.