It is true in general that the set of all invertible elements of a Monoid form a subgroup. The proof is trivial.
However, after some thought, I feel that if we restrict invertible to left or right invertible only, then it does not form a group. It seems so because I cannot imagine a way to prove otherwise.
I am looking for examples of Monoids whose left invertible elements do not form a subgroup or else proof that it does form a group.