The animated objects was not updated for each internal substep for the rigidbody sim.
This would lead to unstable simulations or very annoying clipping artifacts.
I've updated the code to use explicit substeps and tie it to the scene frame rate.
(This is how it is done in the cloth sim for example)
We would also have issues with the simulated objects becoming "magnetic" and sticking together with other general simulation stability issues.
This was solved by updating the bullet library to the latest version and switching to double precision.
I've created pull a pull request for the custom bullet code that we had. So hopefully we can soon build blender with a upstream release.
https://github.com/bulletphysics/bullet3/pull/3018
This fixes: https://developer.blender.org/T47402