On the second page, $c$ is considered an arbitrary positive constant, also $c$ divides $ax+by$ - which means $ax+by=kc$.
If by chance $x/k$ and $y/k$ belong to integers, $c$ will belong to set $S$ and $c$ will be smaller than the least element $d$. Isn't that a contradiction? Where am i wrong?
Suppose that $c$ is an arbitrary positive common divisor of the non-zero integers $a$ and $b$ (trivial if either integer equals $0$) and $\gcd(a,b) = d$. Then $\exists$ $p,q \in \mathbb{Z}$ such that
$$a = pc \qquad b = qc$$
Consequently, we can substitute into $ax+by$ to get
$$(pc)x+(qc)y = d \implies c(px + qy) = d$$
So $px + qy$ is what $k$ is explicitly equal to. Now, if we attempt to do what you propose, we find that
$$a \Big(\frac{x}{px+qy} \Big) + b \Big(\frac{y}{px+qy} \Big) = c$$
If $px + qy = 1$, then there is no issue as it is clear that, in fact, $c = d$ and we do not contradict the minimality of $d$. If, on the other hand, we suppose that $px + qy > 1$ (which would make $c < d$), then the expressions
$$\frac{x}{px+qy} \quad \text{and} \quad \frac{y}{px+qy}$$
are not in the form $X$, $Y$ as mentioned in the comments. In fact,
Question: Is either expression an integer?
To investigate this, let us first note that at least one of $x$ or $y$ must be non-zero, or else $d = 0$. Without loss of generality, let $x \neq 0$ and $x/(px+qy) \in \mathbb{Z}$. Then $\exists \ m \in \mathbb{Z}$ such that
$$x = m(px+qy) \implies x(1-mp) = mqy$$
Rewriting the second expression in the following manner:
$$\frac{y}{px+qy} = \Big(\frac{mq}{mq} \Big) \frac{y}{px+qy}$$
(This is legal since $b \neq 0 \implies q \neq 0$ and $x \neq 0 \implies m \neq 0$). Substituting for $mqy$ gives us
$$\frac{x(1-mp)}{mq(px+qy)} = \frac{x(1-mp)}{xq} = \frac{1-mp}{q} \quad \Big( = \frac{y}{px+qy} \Big)$$
if $y = 0$, then we force $mp = 1 \implies m = p = \pm 1$. Then $a = \pm c \implies b = \pm qa$, contradicting the generality of our choice of the integers $a, b$. Otherwise, if $y \neq 0$ and $y/(px+qy) \in \mathbb{Z}$, then $\exists k \in \mathbb{Z}$ such that
$$1 - mp = kq \implies 1 = p\cdot(m) + q\cdot(k)$$
The equality implies that $p, q$ are relatively prime, for $\gcd(p,q) = 1 \iff$ we can write a $\mathbb{Z}$-linear combination of $p,q$ equal to 1. But then this implies that $c = d$, contradicting both the hypotheses that $c < d$ and $px + qy > 1$. Note that this is true by referring to this question and observing that $a/c = p, b/c = q,$ and
\begin{align}\frac{a}{c} (m) + \frac{b}{c} (k) = 1 &\iff \gcd \Big(\frac{a}{c},\frac{b}{c} \Big) = 1 \\ & \iff c = \gcd(a,b) = d \end{align}
Thus it follows that $c \notin S$.