Mutex locks like this are very slow when you have many cores, especially on multi-socket system where synchronizing is costly. Doing this for all vertices in the radius can make the operation slower than using a single core.

Instead what I suggest is to compute the nearest vertex per PBVH node. And then at the end of this function compare it to the global nearest vertex. For that you can use a spin mutex, since this is a relatively fast operation.

For performance you could also track the nearest_vertex_distance_squared instead of recomputing the length every time, or at least used len_squared_v3v3.

for (int i = 0

len_squared_v3v3(location, vertex) < radius * radius

This test is too slow to perform for every vertex. Is checking if (ss->cache->automask) enough, or can you set some other flag that is quick to check?

Naming conventions: VertexTopologyIterator

Add some comments in this function explaining what the various parts do, for example:

/* Add active vertex and symmetric vertices to queue. */
...
/* Flood fill automask to connected vertices. */

Use less obscure name.

Can we put this condition in a utility function? I see it duplicated in a few places and it's hard to understand.

Above it's (char)i, can we be consistent?

Using a BLI_gsqueue for this is rather slow because of the amount of memory allocations it does, it's not implemented efficiently for this case.

BLI_stack will be more efficient here and the same amount of code.

Commented this already in other code reviews, but would be good to move these 4 lines into a utility function.

It can also be put in sculpt_topology_automasking_init, since it's only needed there, and not when BRUSH_AUTOMASKING_TOPOLOGY is off?

Topology automasking -> Topology Automasking

Automaks disconnected vertices -> Don't affect vertices not connected to the active vertex under the brush