One way to describe your error is that by moving from the $X,Y$-space to the $A,B$-space, you are no longer dealing with the unit square $[0,1] \times [0,1]$ as your possible space for the valid $(X,Y)$ coordinates. You are now dealing with the "upper left diagonal half" of that square as the possible space for your $(A,B)$ coordinates, the triangle formed by the points $(0,0), (1,1)$ and $(0,1)$.
So while your construction of the geometric polygon where the maximum length of the parts is at most $a$ is correct, the incorrect part is that you use that as the probability of it occuring. You are building the quotient of the area of the constructed polygon with the area of all possible values, which you assume is the whole $[0,1] \times [0,1]$ square, with area $1$. But corect would be to use the area of the triangle I described, which has area $\frac12$.
So the geometric area of your polygon is $a^2 - \frac{(1-a)^2}2 - \frac {(2a-1)^2}2$, as you correctly calculated, but the probability is actually double that, as you need to divide by $\frac12$:
$$
\begin{eqnarray}
2a^2 \color{red}{-(1-a)^2} \color{blue}{- (2a-1)^2} &=& 2a^2 \color{red}{- 1 + 2a - a^2} \color{blue}{- 4a^2 + 4a - 1}\\ &=&-3a^2+6a-2 \\ &=& 1 - (3a^2-6a+3)\\ &=& 1-3(a-1)^2
\end{eqnarray}
$$
which is the result in the other answer!
But there is still a hidden assumption made here that turns out to be correct, but is worth pointing out because it may not always be true.
By divding the area of the found polygon by the area of the triangle, we implictely assume that the way we arrive at the points in that triangle makes them uniformly distributed over that triangle.
It is true, but it isn't clear or trivial at all, because we are now no longer dealing with $(X,Y)$ coordinates that come from independent, uniformly distributed random variables $X,Y$. We are know dealing with dependent, non-uniformly distributed random variables $A,B$.
That they are dependent should be clear, if you know that that $B=0.7$, $A$ can't be $0.8$ any more. That is "captured" by the $B \ge A$ condition.
That they are non-uniform can be seen by finding the probability that $B \le \frac12$. Since $B$ is the maximum of $A,B$, this is equivalent of finding the probability that $X \le \frac12, Y \le \frac12$. Since $X$ and $Y$ are independent, that probability is just the product of the separate probabilities, so we get
$$
\begin{eqnarray}
P\left(B \le \frac12\right) &=& P\left(X \le \frac12, Y \le \frac12\right) \\
&=&P\left(X \le \frac12\right) P\left( Y \le \frac12\right)\\
&=& \frac12\frac12 = \frac14
\end{eqnarray}
$$
If $B$ would be uniformly distributed over $[0,1]$, that probability would have to be $\frac12$.
There is a way to transform your solution to work in the 'easy to understand/argue about' X,Y space. Simply divide the $[0,1] \times [0,1]$ square into the "upper left diagonal" triangle from above and the "lower right diagonal" part formed by $(0,0),(1,1)$ and $(1,0)$. Then realize that the upper left diagonal part is where $A=X, B=Y$ and the lower right diagonal part is where $A=Y, B=X$.
Your construction in the upper left diagonal part from above works exactly the same as before, only now you get to apply it again to the lower right diagonal part, where it produces a polygonal area of the same shape (I'm sorry, I'm not that good with making a picture of that).
So now your two "good" areas combined have double the area as before, so this again leads to the same probability as above.
