This page explains why transforming a normal vector from object space to eye space is;
When lighting is enabled in OpenGL, the normal vectors are used to determine how much light is received at the specified vertex or surface. This lighting processing is performed at eye coordinate space, therefore, normal vectors in object coordinates are also transformed to eye coordinates with GL_MODELVIEW matrix.
However, normal vectors are transformed in different way as vertices do. We cannot simply multiply GL_MODELVIEW matrix and normal vectors. Consider a normal vector (1,0,0) for a vertex (0,0,0). If GL_MODELVIEW matrix is simply translating 2 unit up along Y-axis, then the vertex coordinates will be (0,2,0). But, the normal vector should remain same as (1,0,0), not (1,2,0).
In order to understand how normal vectors are transformed to eye space, think the normals as coefficients of plane equations, which are perpendicular to the planes.
Imagine a triangle polygon with 3 vertices; v1-v2-v3, and the normal of this surface is as a homogeneous plane. (For Euclidean space, the normal is .) If we think of the triangle as a homogeneous plane, then the plane equation becomes;
Since 3 vertices lie on this plane, the plane equation is also true when we substitute these vertices to the equation. For example, for , it satisfies;
The equivalent matrix form of the plane equation is;
The plane equation is multiplying the transposed normal (nT) and the vertex (v) together. You can read "The vertex v in object space lies on the plane with the normal ".
Now, we modify the above equation to acquire normal vector transformation formula by inserting GL_MODELVIEW matrix M-1M in between nT and v ;
(Note that the following equation is still equivalent to the above because M-1M is identity matrix.)
As you can see, the right part of the above equation is the vertex in eye space because it is transformed by MODELVIEW matrix M, and the left part is going to be the normal vector in eye space because this plane equation is also transformed.
It reads the above equation as "The transformed vertex lies on the transformed plane in eye space with the transformed normal ".
Therefore, transforming a normal from object space to eye space with GL_MODELVIEW matrix M is;
Or, by converting pre-multiplication to post-multiplication form, we get;
Suppose a normal vector is (0, 0, 1) and the matrix has only translation part, moving to (1, 2, 3). The translation matrix, its inverse and transpose matrix and transformed normal would be;
The normal vector remains unchanged (0, 0, 1) after translation. Therefore, it works as supposed to be if the matrix has translation only. Note that nw component is changed to -3, but it doesn't affect the direction and length of the normal vector.
If the transform matrix contains rotation only, the normal vectors are also rotated same as vertices do. The following is the matrix rotating 30 degree along X-axis, and transforming a normal vector (0, 0, 1) multiplying by the inverse and transpose matrix should become (0, -0.5, 0.866).
Note that the rotation-only matrix is unchanged after invert and transposed because it is an orthogonal matrix. (The inverse matrix is equal to the transpose matrix) Therefore, we don't need to compute the inverse and transpose process if the matrix has rotation only. We can multiply the transform matrix directly for normal vectors.
Consider tramsforming a normal vector (0, 0.6, 0.8) by scaling (2, 4, 5). The transform matrix and the transformed normal with the inverse and transpose matrix follow;
Note that the transformed normal vector is not unit length any more, plus its direction is also changed because of non-uniform scaling. Therefore, we need to re-normalize the normal vector in order to make it unit length.
What would be the result if the scale factor is -1, for example, scaling by (1, 1, -1)? See the visual result here.
Transforming a normal vector from object space to eye space multiplying by would work if the transform matrix contains translation and rotation. However, if the transform has scaling, the normal vector must be re-normalized to make it unit length.
Since a rotation-only matrix is orthogonal (the inverse matrix and transpose matrix are same, so the original matrx and the inverse then transpose matrx are also identical), we can skip the process to invert and to transpose the matrix.
To optimize the performance of normal transformations, it would be better to define a separate matrix for the normal vectors, which is only containing the rotation parts from GL_MODELVIEW matrix (ignoring translation and scaling), and use this matrix to transform normal vectors from object space to eye space without computing inverse and transpose.