I did not figure out, how to setup the dependency on the collection correctly yet. The modifier should execute again, when a object is added or removed from the collection (currently it doesn't).
After discussing this issue in Blender chat, I think we can merge this even with this bug. The boolean modifier and collision collections have the same problem. We'll have to look into properly solving it at some point, but it does not necessarily have to block this patch.
Ref T82372.