Proof that I is an ideal of $R$
By this question, $R = \mathbb{Z} + \mathbb{Z}\frac{(D + \sqrt D)}{2}$.
Hence it suffices to show that $a\frac{(D + \sqrt D)}{2} \in I$ and $\frac{(-b + \sqrt D)}{2}\frac{(D + \sqrt D)}{2} \in I$.
$a\frac{(D + \sqrt D)}{2} = \frac{(aD + a\sqrt D)}{2} = \frac{(aD + ab + a(-b + \sqrt D))}{2} = a\frac{(D + b)}{2} + a\frac{(-b + \sqrt D))}{2}$
Since $D \equiv b^2$ (mod $4$), $D \equiv b^2 \equiv b$ (mod $2$).
Hence $\frac{(D + b)}{2}$ is an integer.
Hence $a\frac{(D + \sqrt D)}{2} \in I$.
$(-b + \sqrt D)(D + \sqrt D) = -bD - b\sqrt D + D\sqrt D + D
= -bD + D + (D - b)\sqrt D = D - b^2 + (D - b)(-b + \sqrt D)$.
Hence
$\frac{(-b + \sqrt D)}{2}\frac{(D + \sqrt D)}{2} = \frac{(D - b^2)}{4} + \frac{(D - b)}{2}\frac{(-b + \sqrt D)}{2}$.
Since $D \equiv b^2$ (mod $4$) and $D \equiv b$ (mod $2$),
$\frac{(D - b^2)}{4}$ and $\frac{(D - b)}{2}$ are integers.
Hence $\frac{(-b + \sqrt D)}{2}\frac{(D + \sqrt D)}{2} \in I$.
QED
Lemma 1
Let $K$ be a quadratic number field.
Let $R$ be an order of $K$, $D$ its discriminant.
Then $R = \{\frac{(x + y\sqrt D)}{2} |\ x \in \mathbb{Z}, y \in \mathbb{Z}, x \equiv yD$ (mod $2)\}$.
Proof:
By this question, every element $\alpha \in R$ can be writen as $\alpha = u + y\frac{(D + \sqrt D)}{2}, u, y \in \mathbb{Z}$.
Hence $\alpha = \frac{(2u + yD + y\sqrt D)}{2} = \frac{(x + y\sqrt D)}{2}$, where $x = 2u + yD$.
QED
Lemma 2
Let $D$ be a non-square integer such that $D \equiv 0$ (mod $4$) or $D \equiv 1$ (mod $4$).
By this question, there exists an order $R$ of a quadratic number field $K$ such that the discriminant of $R$ is $D$.
Let $ax^2 + bxy + cy^2$ be an integral binary quadratic form whose discriminant is $D$.
By what we have proved in the above, $I = \mathbb{Z}a + \mathbb{Z}\frac{(-b + \sqrt{D})}{2}$ is an ideal of $R$.
Let $(R : I) = \{ z \in K |\ zI \subset R \}$.
Then $(R : I) = \mathbb{Z} + \mathbb{Z}\frac{(b + \sqrt D)}{2a}$.
Proof(based on Cohen's A course in computational algebraic number theory):
Since $D$ is non-square, $a \ne 0$.
Let $z \in (R : I)$
Then $za \in R$.
By Lemma 1, there exist integers such that $za = \frac{(x + y\sqrt D)}{2}$ and $x \equiv yD$ (mod $2$). Then $z = \frac{(x + y\sqrt D)}{2a}$
$z\frac{(-b + \sqrt{D})}{2} = \frac{(x + y\sqrt D)}{2a}\frac{(-b + \sqrt{D})}{2}
= \frac{(-bx + x\sqrt D - by\sqrt D + yD)}{4a} = \frac{(-bx + yD + (x - by)\sqrt D)}{4a} \in R$.
By Lemma 1, $x \equiv by$ (mod $2a$).
Hence there exists an integer $u$ such that $x = by + 2au$.
Then $z = \frac{(x + y\sqrt D)}{2a} = \frac{(2au + y(b + √D))}{2a} = u + y\frac{(b + \sqrt D)}{2a}$. Hence $z \in \mathbb{Z} + \mathbb{Z}\frac{(b + \sqrt D)}{2a}$.
Hence $(R : I) \subset \mathbb{Z} + \mathbb{Z}\frac{(b + \sqrt D)}{2a}$.
It remains to prove the opposite inclusion.
Let $\gamma = \frac{(b + \sqrt D)}{2a}$.
Since $\mathbb{Z} \subset (R : I)$, it suffices to prove that $\gamma \in (R : I)$.
Since $D \equiv b$ (mod $2$), $\gamma a = \frac{(b + \sqrt D)}{2} \in R$ by Lemma 1.
On the other hand, $\gamma\frac{(-b + \sqrt{D})}{2a}
= \frac{(D - b^2)}{4a} = -c \in R$.
Hence $\gamma I \subset R$.
Hence $\gamma \in (R : I)$.
QED
Proof that $I$ is invertible if and only if $ax^2 + bxy + cy^2$ is primitive
The idea of the following proof is borrowed from Cohen's A course in computational algebraic number theory.
We use the following notation.
Let $x_1, \cdots, x_n$ be a finite sequence of elements of $K$.
We denote by $[x_1, \cdots, x_n]$ the subgroup of $K$ generated by $x_1, \cdots, x_n$.
By this question, $I$ is invertible if and only if $I(R : I) = R$.
By Lemma 2, $(R : I) = [1, \frac{(b + \sqrt D)}{2a}]$.
Hence $I(R : I) = [a, \frac{(-b + \sqrt D)}{2}][1, \frac{(b + \sqrt D)}{2a}]=
[a, \frac{(b + \sqrt D)}{2}, \frac{(-b + \sqrt D)}{2}, \frac{(D - b^2)}{4a}]=[a, \frac{(b + \sqrt D)}{2}, \frac{(-b + \sqrt D)}{2}, c]=[a, b, c, \frac{(-b + \sqrt D)}{2}]=$ [gcd$(a, b, c), \frac{(-b + \sqrt D)}{2}]$.
Hence if $I(R : I) = R$, gcd$(a, b, c) = 1$.
Conversely suppose gcd$(a, b, c) = 1$.
Since $D \equiv b$ (mod $2$), $\frac{(-b - D)}{2}$ is an integer.
Hence
$I(R : I) = [$gcd$(a, b, c), \frac{(-b + \sqrt D)}{2}]
= [1, \frac{(-b + \sqrt D)}{2}]
= [1, \frac{(-b - D)}{2} + \frac{(D + \sqrt D)}{2}]
= [1, \frac{(D + √D)}{2}] = R$
