I'm making a 2.5D game engine (in JavaScript) and I'm trying to get it to detect whether a block is under the cursor. This would be a simple task were it restricted to single-level isometric coordinates, but instead it's supposed to support isometric worlds, trimetric worlds and well, any projection angle really, in addition to it being multi-level (tiles can be stacked on each other up to an arbitrary limit). Almost all the conditions are arbitrary, but I've managed to narrow the problem to this, for now:
I need to know whether a point is inside a hexagon:
What I've set up so far is a bit hackish, I'm checking whether the point is inside the bounding rectangle first to drop out the most obvious cases. Next, I'm checking whether it's inside the rectangle formed by the points ACDF, another easy check. Now the hard part is that I need to check whether the point is inside the triangles. That's pretty straightforward if you have the barycentric coordinates, but I seem to have lost all the knowledge of how to obtain them.
Do take into account that while the inner rectangle is always a rectangle, the triangles might not be equal-sided. Bonus points if there's an optimization to be made because the bottom (or top, respectively) of the triangle is flat.