This option was using the dot product between the view normal and the
vertex normal to mask vertices. This was causing artifacts and affecting
the shape of the brush tip on curved and irregular surfaces, as well as
artifacts in areas with high frequency detail.

Now it uses a weighted average of the vertex normal and its neighbors
with a smoothstep falloff in a defined area. It still produces some
artifacts with highly deformed topology, but the results are much better
in general.

I also renamed and cleaned up the function using the sculpt API in order
to move this functionality to the automasking system in a later
refactor.

Before

After