This is kind of a crazy idea, but I want to write it down so I don't forget -- I might delete this community wiki answer later. (In particular, it being CW means that upvotes/downvotes to this question don't mean anything.)
Anyway, the primary idea, which may be incorrect, is the following:
the characterizing property of direction is that it is scale-invariant.
This is essentially how direction is defined in the case of vector spaces, so it should probably be the idea in other contexts as well.
Here is how this idea can be implemented in the case of an arbitrary metric space (this definition is similar to that for vector spaces, but does not always coincide -- especially when the vector space is not even metrizable).
Given a metric space $(X,d)$, for each point $x \in X$, let $Sim(X,x)$ denote the local similitude group centered at $x$ (the group of all surjective, equivalently bijective, similitudes which fix $x$).
By convention, with respect to itself, the point $x$ has no direction.
Then we say that, for each point $p$ in $X \setminus x$, the direction of $p$ relative to $x$ is the orbit of $p$ under the action of $Sim(X,x)$. (This is an equivalence class of points in $X$.)
(Being an orbit of $Sim(X,x)$ gives both scale-invariance and the result that the direction is relative to a certain point, since the elements of $Sim(X,x)$ are exactly those which "scale" the space $X$ by a positive constant while still fixing the point $x$, and per definition an orbit is invariant under the action of a group.)
I believe that this definition coincides with the usual one for $\mathbb{R}^n$ with the Euclidean metric -- I don't know if it coincides with the definition for Riemannian manifolds using geodesics instead of the tangent spaces.
On the other hand, I kind of like this definition because it seems very Kleinian.
Update: The above is (almost) COMPLETELY wrong.
In particular, we don't want directions to be invariant under rotations or reflections. However, rotations and reflections correspond to the non-trivial elements of the local isometry groups $Iso(X,x)$ which are obviously subgroups of the local similitude groups $Sim(X,x)$.
So, the "direction" as defined above would be invariant under the action of non-trivial elements of $Iso(X,x)$, which it inherently should not be.
I was confusing $Sim(X,x)$ with something like a "group of pure scalings centered at $x$" ("group of dilations centered at $x$"), which, if and when such a thing could be defined, would clearly be a subgroup, but in most cases a proper subgroup, of $Sim(X,x)$. So the definition above is too coarse of an equivalence relation to successfully define a notion of direction.
My guess is that, if and when a "pure scaling" or "dilation" group exists, denote it $Dil(x)$, then one would have that $$Sim(X,x)/Iso(X,x) \cong Dil(x) \,. $$
This is motivated by the example of Euclidean space, where $Sim(X,x)$ is the conformal group, $Iso(X,x)$ is the orthogonal group, and the group of dilations is isomorphic to $\mathbb{R}_{>0}$.
Now, obviously in the case of real vector spaces, a dilation group always exists, and it is always isomorphic to $\mathbb{R}_{>0}$. Therefore, the following definition generalizes the well-known definition/notion of direction used in real vector spaces.
By convention, the point $x$ has no direction with respect to itself. For any point $p \in X \setminus \{x\}$, the direction of $p$ relative to $x$ is the orbit of $p$ under the action of $Dil(x)$.
Again, since usually $Dil(x) \subsetneq Sim(X,x)$, this definition is strictly different than the originally proposed one.
Anyway, this is an even better definition (I think), because (1) it's still very Kleinian, and (2) it leads to a natural classification of geometric structures -- angle measures are the natural invariants of $Sim(X,x)$, rotations and reflections are the actions of $Iso(X,x)$, and directions are the natural invariants of $Dil(x) \cong Sim(X,x) / Iso(X,x)$, in addition to, of course, distance being the natural invariant of $Iso(X)$.
