When there are multiple depsgraphs (for example the main one + a separate one for rendering or VSE precaching) and a driver is removed, only the main depsgraph gets told to reconstruct its relations. This causes evaluation of the by-now-removed driver in the other depsgraphs.
A proper fix would be to get the background jobs to refresh their depsgraph as well (or to abort/restart), but that's too big of a change for Blender 3.0.
This at least fixes the crash.