How do game engines avoid recalculating normals upon mesh rotation?

Question

Assume that a face of a mesh is defined by vertices A, B, C and that the normal of the face is defined by (B - A) x (C - A).

Rotating the mesh about its local origin with matrix R makes the normal calculation R(B - A) x R(C - A).

To my knowledge there is no algebraic property of the cross product that lets us simply do R((B - A) x (C - A)) to rotate the normal just like all the vertices. The cross product preserves distribution over an orthonormal matrix.

Instead it seems like the normal needs to be entirely recalculated every time the object is rotated. This seems unreasonably expensive for a realtime application so I was curious if game engines have some way to efficiently transform normals, approximate them, or avoid them altogether.

Answer 1

They don't.

When you rotate a mesh, the normals in memory are not changed in any way. All that changes is the object's transformation matrix is updated. The raw mesh vertex and normal data remains unchanged, and can be shared between many instances of the model that get drawn with different positions, orientations, and scales.

These transformations are applied "just in time" during rendering. We need to multiply the vertex positions by the camera's view and projection matrices anyway to figure out where to draw it on screen, so we can just pre-multiply those with the object's transformation matrix (often called either the model matrix or the world matrix) to get a combined "MVP" matrix (model-view-projection), then transform each vertex with just a single matrix multiplication in the vertex shader, as it's being drawn.

The same trick works for normals, it's just that we use the inverse-transpose of the model matrix multiplied by the view matrix (no projection) to map the normal into view space (where we usually want it in deferred rendering, though some lighting systems use a different coordinate space for lighting).

This isn't unduly expensive. A matrix multiplication per vertex isn't breaking the bank when we can do hundreds of them in parallel on modern GPUs, and we'd need to pay that cost anyway every time the camera moves, which in many games is already happening every frame. The ability to combine multiple matrix transformations into a single matrix lets us bake the model's transformation in "just in time" at no added cost per vertex.

For things like CPU-side physics calculations, we can often use the inverse of the model matrix to transform the calculation into the model's local space, rather than transforming all vertex positions and normals into world space. Then once we know which face has been collided with, we can take the one resulting collision normal and transform just it into world space, without having to touch any of the others.

Answer 2

DMGregory's answer explains how this is actually handled, but I'd like to point out a couple of misconceptions:

Assume that a face of a mesh is defined by vertices A, B, C and that the normal of the face is defined by (B - A) x (C - A).

Note that this is only true for “flat shading”, where each triangle forms a visible facet, near-uniformly lit. Most meshes today have normals explicitly defined per vertex, which allows them to be interpolated to simulate curved surfaces. The normals will typically be similar to the cross product, but not equal — in particular, the 3 vertices of a triangle will have different normals, unless the surface the mesh is approximating is actually flat.

These normals cannot be calculated from the mesh, but rather are created at the same time as the mesh, and the algorithm (or author's manual operations) will depend on the intent. For example, a mesh which has the form of a hexagonal tube might be intended to be in fact hexagonal (and the normals will be perpendicular to the faces) or it might be intended to be a low-polygon approximation to a cylinder or wire (in which case the normal of each vertex will point radially outward, so faces have varying normals, and lighting is smooth across the edge between faces).

For game graphics purposes, you should understand normals as a property vertices have, which we wish to preserve the meaning of under various transformations; not as something you can derive from the mesh except when you are making the particular aesthetic restriction of using only flat shading.

“Face normals” are not used or computed at all (unless you are doing something unusual. Normals are interpolated per-pixel across the surface of a triangle, to approximate curved surfaces.

To my knowledge there is no algebraic property of the cross product that lets us simply do R((B - A) x (C - A)) to rotate the normal just like all the vertices.

Yes and no. The matrix can be such that R((B - A) x (C - A)) ≠ R(B - A) x R(C - A), but in that case, it is not a rotation matrix — it is some other less restricted sort of matrix.

But, since normals are not in practice defined by cross products of vertex differences, the cross product doesn't matter. What does matter is that the vertex normals are transformed in a way which matches the transformation of the shape the mesh forms, so they remain appropriate.

In the simple case of a pure rotation matrix, which does not change the angles between vectors, it suffices to rotate the normals with the same matrix. Picture the normals as pins sticking out of the vertices, and understand that we are rotating — or more generally, performing a rigid transformation — of the entire object as a whole.

More generally — such as if the matrix contains non-uniform scale — it is necessary to use a different matrix to transform the normals (as DMGregory's answer mentions), and that is what is in fact done in general-purpose renderers.

Answer 3

A few words on why people use the inverse transpose when mapping normals: suppose your normal at a point \$p_1\$ on the untransformed object is \$n\$ and \$p_2\$ is a nearby point on the object. Suppose also that you are applying the affine transformation matrix \$M\$ to your object, so that \$Mp_1\$ and \$Mp_2\$ are the new locations of \$p_1\$ and \$p_2\$ on the transformed object.

You want to transform the normal using a matrix \$N\$ so that the angle between \$n\$ and \$p_2 - p_1\$ is preserved:

$$ \begin{align} (Nn) \cdot (Mp_2-Mp_1) &= n \cdot (p_2 - p_1) \\ n^T N^T M (p_2 - p_1) &= n^T (p_2-p_1) \end{align} $$ and this equation is true for any choice of \$p_2\$ if \$N^TM\$ is the identity, i.e. \$N = M^{-T}\$.