The OptiX SRT motion expects a motion defined by translation, rotation, shear and scale, but the matrix decomposition code in Cycles was not able to extract shear information and instead produced a stretch matrix with the information baked in. This caused conflicting transforms between traversal and shading and lead to render artifacts.
This patch changes the matrix decomposition to produce factors inline with what OptiX expects to fix that. It should also address a similar concern for Embree (with D6575), since it uses the same factors.

I verified that the algorithm generates correct factors by doing this at the end of transform_decompose:

Transform T = transform_identity();
T.x.w = decomp->y.x;
T.y.w = decomp->y.y;
T.z.w = decomp->y.z;
Transform R = M;
Transform S = transform_identity();
S.x.x = decomp->y.w;
S.y.y = decomp->z.w;
S.z.z = decomp->w.w;
S.x.y = decomp->z.x;
S.x.z = decomp->z.y;
S.y.z = decomp->w.x;

Transform test1 = T * M * S; // Recompose matrix
Transform test2 = *tfm; // Original matrix
assert(test1 == test2);

All the Cycles unit tests complete successfully still too (including the motion blur ones).