How to preserve perspective of 3D image while altering "pitch" and "bearing"

Question

This is sort of a mixed CS/Math problem. Hopefully this is a good place to ask -- the question strikes me as primarily geometric. If not, please let me know how I might best find an answer to the problem. Also, hopefully the tags are helpful; I'm not 100% comfortable with what some of these terms mean, but they seemed the most relevant.

I'm a programmer, attempting to overlay a (seemingly) 3D image of a building over its footprint on a map (using Mapbox).

The basic idea is that you provide an image and the coordinates of its four corners, which isn't hard to get to line up correctly so that, from a certain angle, the building image looks like it's standing vertically in the space of the building.

Here's an example building image.

The trouble is, if someone alters the pitch or bearing of the map, things no longer line up.

Here's an example of the problem.

By modifying the corners of the image (bounded by the red line), it's possible to get the image to more or less line up correctly, though I haven't been able to achieve a 100% match, because changing any one corner tugs the rest of the image into misalignment, in a pretty much endless loop.

The question:

Is there a way to calculate, based on bearing (degrees of map rotation from facing pure north) and pitch (degrees of up or down rotation off of a direct vertical perspective), where the corners of the image should be to preserve realistic angles of the building walls in the image?

Some caveats/thoughts/additional questions:

• Is this even possible? Or would I effectively need a different image for each bearing/pitch combo in order to effectively simulate a 3D structure from every angle.

• The building is not necessarily a square (or even a parallelogram). The image is always a rectangle though.

• Similar to the above, this solution has to be generalizable to an infinite number of differently shaped buildings (in rectangular images).

And just to state the hopefully obvious, I'm not trying to get others to do my "homework" and am happy to do the calculations on this myself. It's just been decades since I took even basic geometry, and I don't have any idea where to start on this one.

Answer 1

You can't get a 3D shape to look correct at all angles. What you can do is get a single flat surface to look correct if you adjust the image corners depending on the location of the observer, and apply a projective transformation to the image as a whole.

So here is what I'd do. Start by creating one image for each face of your building, with a transparent background. If it has some complicated shape, you may have to simplify, and hope noone will notice (too much). Using different sets of images might help here.

Now you have an image of a face of the building, and the 3d coordinates of the corners of the building in real world. Also the coordinates of the observer in the real world. Connecting observer to each of the corners yields lines of sight for these. The points where these lines intersect the map are the points where you have to place your corners of the building if it is to appear correct.

So now you have the 2d in-plane coordinates of the building face corners. But the corners of the building are not neccessarily the corners of the picture. (And if corners are problematic, you can take any other set of four non-collinear points which you can easily get coordinates for). So you might want to follow this guide of mine and compute the transformation matrix between face image (texture?) and map plane. Use that to map the corners of the picture from their coordinate system to the map plane.

Since I'm assuming a conversion between a flat map in the ground plane and a 3d surface in real world, the actual viewing direction of the camera doesn't enter this setup. It doesn't matter which direction the camera looks, as long as the position of the camera remains the same. Your statement “if someone alters the pitch or bearing of the map, things no longer line up” appears to contradict this claim. So perhaps I misunderstood some aspect of your question? If so, please clarify.

(I once had the pleasure of doing something like this in real life, if only for a single point of view.)

Answer 2

For each image of a building

• Determine all "corners", all "edges" between the corners, and all "faces" surrounded by edges. For simplicity, you may even want to refine this into a triangulation.
• For each corner, find its true $(x,y,z)$ coordinates.
• Depending on bearing and pitch and position (i.e., your display perspective), determine the 2D coordinates for each corner.
• For each (triangular) face, you obtain an (affine) linear map 2D$\to$2D that is determined by mapping the 2D vertices of the image via the corresponding 3D coordinates to 2D coordinates in the display. Apply this affine map to the triangular area enclosed by the three vertices.

In the last step you may run into the problem of generating walls (from their backside) when they should be hidden. These can be detected (and eliminated) by checking the sign of the determinant of the affine map. But you may run into even more complex hidden wall problems if the building is not convex. In that case you really need to look into more complex rendering algorithms.