Let $f:[0,1]\to (0,1)$ be a homeomorphism. Let us suppose, for a contradiction, that $f$ is not monotonic. In particular, either there exists:
(1) $a<b<c$ with $f(a)<f(b)$ and $f(b)>f(c)$
or
(2) $a<b<c$ with $f(a)>f(b)$ and $f(b)<f(c)$.
Let us consider case (1) without loss of generality.
Exercise 1: Prove that the image of $(a,c)$ under $f$ is not open in $(0,1)$. In particular, we have a contradiction and $f$ must be monotonic.
Let us assume without loss of generality that $f$ is monotonically increasing.
Exercise 2: Prove that $f$ is not surjective. In particular, we have a final contradiction.
Therefore, $f$ is not a homeomorphism.
The following exercises are relevant:
Exercise 3: Prove that there is a continuous surjection $f:(0,1)\to [0,1]$. Do you think that there is a continuous surjection $f:[0,1]\to (0,1)$?
Exercise 4: Does there exist a surjective open map (i.e., open sets are mapped to open sets) $f:[0,1]\to (0,1)$. Do you think that there is a surjective open map $f:(0,1)\to [0,1]$?
Exercise 5: Find an example of a continuous bijection between topological spaces that is not a homeomorphism.