Your intuition is good, the answer is the median of $(a_i)_{1\leq i\leq n}$.
Let $f_1(x)=\sum_{i=2}^{n-1}\vert x-a_i\vert$. Then you have, for all $x$:
$$f(x)=\vert x-a_1\vert + f_1(x)+\vert x - a_n\vert$$
Considering this equation, you can show that $f_1$ et $f$ have the same minimum, noticing that:
- the minimum is not outside $[a_1,a_n]$
- $\forall x\in\lbrack a_1,a_n\rbrack$, $f(x)=f_1(x)+(a_n-a_1)$
Repeating this, you can "forget" extremal points at each step. At the end you only have one or two points left. Then you just have to consider the two cases: $n$ even and $n$ odd.