Normal vectors represent the direction a surface is facing. Normals are perpendicular to the surface tangent. Normals are often created when building a model and stored in the model file.

Creating triangle normals

For triangles, normals are created from the triangle vertex positions. Since the normal vector is derived from the vertex positions, the order the vertices 'wrap' around the triangle face affects normals. Forming vectors from the vertex positions results in edge vectors. The edge vectors lie in the plane of the triangle face and are tangent to the face. So, the cross product of two edge vectors results in a vector perpendicular to the tangent: the normal vector.

We will create normals according to the right-hand-rule. The side of the triangle where the vertex order is counter-clockwise is the 'front' of the triangle and the normal vector will point away from this side. Given a triangle with vertices \(v\) ordered 0-2, the normal can be created:

$$ \mathbf{a} = \mathbf{v_1} - \mathbf{v_0} \mathbf{b} = \mathbf{v_2} - \mathbf{v_1} \mathbf{n} = \mathbf{a} \times \mathbf{b} $$

Normal matrix

Normals should be rotated with the model. However, they should not be translated, scaled, or sheared. Avoiding translation transformations can be done by dropping the homogeneous dimension used for translation and restricting normal transforms to regular 3D space. However, we still must account for transforms in the \(3 \times 3\) portion of the matrix that incorrectly transform normals. If a model has been transformed by a matrix \(\mathbf{M}\), then the normals should be transformed by another matrix \(\mathbf{N}\) to ensure they are perpendicular to the surface tangent.

We can create \(\mathbf{N}\) beginning with these observations:

$$ \mathbf{n}^T\mathbf{t} = 0 $$

$$ \mathbf{t}_M = \mathbf{Mt} $$

$$ \mathbf{n}_N = \mathbf{Nn} $$

$$ \mathbf{n}_N^T \mathbf{t}_M = 0 $$

$$ \mathbf{n}^T\mathbf{t} = 0 $$

$$ \mathbf{n}^T\mathbf{I}\mathbf{t} = 0 $$

$$ \mathbf{n}^T\mathbf{M}^{-1}\mathbf{M}\mathbf{t} = 0 $$

$$ (\mathbf{n}^T\mathbf{M}^{-1})(\mathbf{M}\mathbf{t}) = 0 $$

$$ \mathbf{t}_M = \mathbf{Mt} $$ can be inserted into the equation:

$$ (\mathbf{n}^T\mathbf{M}^{-1})\mathbf{t}_M = 0 $$

$$ \mathbf{n}_N^T \mathbf{t}_M = 0 $$ the other term must be the transformed normal:

$$ \mathbf{n}^T_N = \mathbf{n}^T\mathbf{M}^{-1} $$

$$ (\mathbf{n}^T_N)^T = (\mathbf{n}^T\mathbf{M}^{-1})^T \ \mathbf{n}_N = (\mathbf{M}^{-1})^T \mathbf{n} $$

$$ \mathbf{N} = (\mathbf{M}^{-1})^T $$

The normal matrix is the regular transform's inverse transposed. This can be created in GLM using glm::inverseTranspose