The matrix interpolation function interp_m3_m3m3() decomposes the matrices into rotation and scale matrices, converts the rotation matrices to quaternions, SLERPs the quaternions, and converts the result back to a matrix. Since quaternions cannot represent axis flips, this results in interpolation problems like described in T77154.

Our interpolation function is based on "Matrix Animation and Polar Decomposition", by Ken Shoemake & Tom Duff. The paper states that it produces invalid results when there is an axis flip in the rotation matrix (or negative determinant, or negative scale, those all indicate the same thing). Their solution is to multiply the rotation matrix with -I, where I is the identity matrix. This is the same as element-wise multiplication with -1.0f. My proposed solution is to not only do that with the rotation matrix R, but also with the scale matrix S. This ensures that the decomposition of A = R * S remains valid, while also making it possible to conver the rotation component to a quaternion.

There is still an issue when interpolating between matrices with different determinant. As the determinant represents the change in volume when that matrix is applied to an object, interpolating between a negative and a positive matrix will have to go through a zero determinant. In this case the volume collapses to zero. I don't see this as a big issue, though, as without this patch Blender would also produce invalid results anyway.

This patch contains two commits. The first adds the unittests for the current implementation. The second contains my proposed fix and of course more tests for it. It is my intention to keep those commits separate.

Below is a demonstration of interpolation between two matrices with Scale X = -1. The only difference between the matrices is the translation. The left-hand object has a Copy Transform constrant, targeting the right-hand object. The purple sphere shows the influence slider value for that constraint.