I tried to play around to find solutions to this, and it seems like I had "Persistent Data" enabled under render settings. After disabling it the render times increased but I no longer had these issues.

I've been using Blender 3.0 Betas and the latest stable release and on my project I'm experiencing this issue daily.