If one has any binary operation $\ast$ on the given set $S$ with an identity $1$ for which any nonidentity element $a$ is its own inverse, then $\ast$ cannot define a group structure on $S$: If it did, the subgroup $\langle a \rangle = \{1, a\} < S$ generated by $a$ would have $2$ elements, but any subgroup of a group structure defined on $S$ has order dividing $5$. (By the same reasoning, the same is true for any set with an odd number of elements.)
Now, we're asked to exhibit an explicit product $\ast$ satisfying the given hypotheses. The products $1x$, $x1$, $x^2$, $x \in S$, are already determined by the hypotheses, leaving $12$ entries unspecified in the multiplication table of $\ast$. The hypotheses together only impose one condition on the remaining entries: Since the cancellation rule holds, each element in $S$ must appear exactly once in each row and each column of the table. Now, we must have $ab = c$ or $ab = d$, and by relabeling if necessary, we can assume the former, which forces $ac = d$, $ad = b$, $cb = d$, and $db = a$.
These in turn force other relations, and after a few iterations this determines the entire table, so in fact there are only two products $\ast$ that satisfy the conditions (determined by the assignment of $c$ or $d$ to $ab$), and these are isomorphic. As you say, this multiplication must be nonassociative; indeed we have $(ab)d = cd = a$ but $a(bd) = ac = d$.
Since this example is both small and unique, it would be interesting to know if this structure could be described in some way more suggestive than a multiplication table. I've posed this as a new question.
Remark The identity and cancellation rules mean we are in particular looking for an operation $\ast$ that defines a loop structure on $S$. This example is minimal in the sense that all loops with $< 5$ elements are actually groups. Up to isomorphism there are exactly $6$ loop structures on a set of $5$ elements (of course, only one of them, $(\mathbb{Z}_5, +)$, is a group structure).