First, the derivative can be rewritten as a difference quotient. Let's do it step by step.
Let $f(x)=x$ be a function and derive it by its only dependent $x$. The derivative $\frac{d}{dx}f(x)$ of a function $f(x)$ can be interpreted as the slope of the function at a point $x$.
Let's do that visually by drawing the graph of the function $f(x)=2x+1$. Now draw a little triangle below (or above) the graph between $x_1=1$ and $x_2=2$, thus between $f(x_1)=3$ and $f(x_2)=5$.
Now use this triangle to calculate the slope of $f(x)$, which gives you the difference quotient:$$\frac{f(x_2)-f(x_1)}{x_2-x_1}$$
But we do not want the averaged slope between $x_1$ and $x_2$, but the slope in a single point $x$.
The idea is, to let the difference of $x_2$ and $x_1$ become very small (colloquial: zero). So, the very general form of a difference quotient is $$\frac{f(x+\delta x)-f(x)}{\delta x}$$
where $\delta x$ is the distance between $x_1$ and $x_2$ and thus $\delta x \rightarrow 0$.
And this difference quotient is nothing else but the derivative of $f(x)$ derived by the variable $x$. And the origin of the '$d$' in $dx$ becomes clear, as it is the $\delta$ in $\delta x$. (with partial derivatives you usually do not write '$d$' but '$\delta$')
Second, with an integral, you do it the other way round. Finding the function $f(x)$ when you've got its derivative $f'(x)$. Thus, you are not calculating a difference quotient any more and thus you do not have $\frac{d}{dx}$.
You write $dx$ at the end of an integral, to show, that the function is integrated by the variable $x$. This becomes very important, when you deal with functions, which are dependent on more than one variable.
(Edit: Ha01 gave a better explanation on this.)
Third, when you derive our function $f(x)=2x+1$, it gives you $f'(x)=2$. The constant term $1$ disappears. Now take another function $g(x)=2x+5$. Its derivative $g'(x)$ is as well constant $2$.
When you calculate the anti-derivative, you have to take into account that any constant term will disappear with every derivative. And thus not to specify one single anti-derivative, you denote the constant term $C$ to describe a family of functions.