I was playing around in $\mathsf{Set},$ trying to reduce it modulo isomorphisms to make a category $\mathsf{Card},$ letting the objects of $\mathsf{Card}$ be the isomorphism classes of $\mathsf{Set}$ and let the morphisms of $\mathsf{Card}$ be the isomorphism classes of $\mathsf{Set}^\to$ (the arrow category). Unfortunately, I got stuck trying to define composition in the obvious way.
Here's the approach I was taking. Given $a\in\mathsf{Set}$ or $f\in\mathsf{Set}^\to,$ we denote their respective isomorphism classes by $|a|$ and $\bar f.$ I've shown that each $\bar f$ uniquely determines a source and a target $|a|$ and $|b|,$ by taking any $f\in\bar f,$ and letting $a,b$ the source and target of $f$ (this is independent of our choice of $f$). If we have $\bar f:|a|\to|b|$ and $\bar g:|b|\to|c|,$ then there exist $a\in|a|,b\in|b|,c\in|c|,f\in\bar f,$ and $g\in\bar g$ such that $f:a\to b$ and $g:b\to c.$ It seems natural to define $\bar g\bar f:=\overline{gf},$ but I'm having trouble showing independence from the choices of $a,b,c,f,$ and $g.$
I started by taking $f_j:a_j\to b_j$ and $g_j:b_j\to c_j$ for $j=1,2,$ and taking isomorphisms $\langle u_1,v_1\rangle:f_1\to f_2$ and $\langle u_2,v_2\rangle:g_1\to g_2.$ So, $f_2u_1=v_1f_1$ and $g_2u_2=v_2g_1.$ Now, if I could find some iso $u:a_1\to a_2$ such that $\langle u,u_2\rangle:f_1\to f_2,$ then I'd be done. Likewise if I could find an iso $v:c_1\to c_2$ such that $\langle v_1,v\rangle:g_1\to g_2.$ Now, the latter doesn't seem feasible, since there's no guarantee that $v_2$ should map fibers of $g_1$ bijectively to fibers of $g_2.$ I've not had any success demonstrating the former, either.
If I use the Axiom of Choice, then I can show that if $f:A\to B$ and $g:X\to Y$ are isomorphic objects in $\mathsf{Set}^\to$, then for any iso $v:B\to Y,$ there is an iso $u:A\to X$ such that $\langle u,v\rangle:f\to g.$ From there, I can finish the proof that the operation is well (and uniquely) defined. In fact, this result seems to imply the Axiom of Choice, as well, which makes me suspect that it is necessary. If so, would someone be able to outline a proof or provide a reference?
If not, then could someone help me get "unstuck"?
Now, if we stick to isomorphism classes of injective functions, we can categorify the cardinals as a partial order (a well-order iff Choice holds), but I'd like to include more isomorphism classes than that, if possible.
Added: As Eric points out, my difficulties are only to be expected. It would seem, then, that my desires might be fruitless, and that only isomorphism classes of injective functions allow such composition to be well-defined. Am I correct?