Well, if we just pull all the things together, the pile may look confusing. A system could help to bring in an order. There may be no best way to do this, and everyone would organize these concepts to their own taste and experience. I will try to share my way of seeing these concepts. Please consider these notes as an informal overview. The precise definitions and statements you can find, for instance, in the references to this answer.
A smooth manifold possesses something which we call a smooth structure. This structure gives rise to vector fields, differential forms, tensor fields, the Lie derivatives, the exterior derivative on forms, - and all these things age given to us naturally, that is they don't require any choice whatsoever to be defined. Following this direction, one comes up with the idea of natural (vector) bundles and natural differential operators on manifolds.
Lie derivatives and the exterior derivative are some examples of natural differential operators. Tensor bundles, jet bundles give us examples of natural bundles.
In the real life applications, we deal with manifolds equipped with an additional structure, which is usually a choice of some section of a natural vector bundle, or a differential operator between natural bundles. These choices are made with certain nice behavior of the resulting structure in mind.
For instance, we choose a positive-definite symmetric covariant tensor of valence 2 and call it a Riemannian metric. A symplectic structure is a choice of a non-degenerate antisymmetric covariant tensor of valence 2. There are many other examples.
When we make choice, we usually expect the abundance of options to choose from. This is the case with Riemannian metrics, connections, but symplectic structures are only available on even-dimensional manifolds (trivially). There are more sophisticated examples, of course.
If we choose a connection on a manifold, there may not much to say about it, but one thing is there: a parallel transport. In fact, choosing a connection or a parallel transport are equivalent to each other (the standard exercise).
As I mentioned in the beginning, there are natural differential operators on manifolds, namely Lie derivatives and the exterior derivative. The issue with the exterior derivative is it is only defined on antisymmetric covariant tensors (differential forms). The Lie derivative $\mathfrak{L}_X$, on the other hand is defined on all tensor fields, but it "is not tensorial in the $X$ slot", so the iterated application of it is somewhat complicated. And yes, the iterated application of the exterior derivative vanishes: $\mathrm{d} \circ \mathrm{d} = 0$.
Connections are tensorial in the $X$ slot, so we can more easily iterate their action (at least, on tensor bundles), and this gives us, among other things, a notion of the curvature operator of the connection. I recommend this answer, if you want to make these things precise.
The Koszul formula (see P.Petersen, Riemannian geometry, Chapter 2) can be interpreted as the calculation of the Levi-Civita connection in terms of the Lie derivative, the exterior derivative, and the Riemannian metric:
$$
2 g( \nabla_Y X, Z) = (\mathfrak{L}_X g) (Y,Z) + ( \mathrm{d} \theta_X ) (Y,Z)
$$
where $\theta_X (Y) = g(X,Y) $, or, in other terms, $\theta_X = X^{\flat}$.
To summarize, given a Riemannian metric $g$ on a (smooth) manifold, you get a uniquely defined (by a formula!) connection, with respect to which your metric is parallel: $\nabla g = 0$. This gives you the Riemannian tensor calculus, and lots of other useful things, known collectively as Riemannian geometry.
Of course, one can do similar things with other kinds of structure, and develop other geometric disciplines.