Flamegraph by @Jacques Lucke (JacquesLucke):

It shows that most time is spent in the filtering, especially in BLO_library_path_explode(). This seems like a cheap function but turns out it does a file system query. This code path is called a lot, so this ended up being a significant bottleneck.

The BLO_library_path_explode() call should actually not be needed, we already have all the information available. This also makes the preceding path joining unnecessary which also shows up in the flamegraph for a fair amount of time.

---

Note that this could be optimized further, but at least it's keeping Blender in a much more usable state now.