For me it only happens when I toggle / uncheck the highest collection in the hierarchy by clicking on the "eye"-symbol. Toggling the seperate collections via keyboard (1,2,3,...) is surprisingly not causing the issue, only clicking by mouse.
My setup: