This patch adds an attribute API to the sculpt code. I went through many iterations of this in sculpt-dev; this is the result. Note that this patch is related too and based off of D14272.
The idea is to have a unified way of creating temporary or permanent attributes across PBVH_FACES, PBVH_GRIDS and PBVH_BMESH.
Basic Design
The sculpt attribute API can create temporary or permanent attributes (only supported in PBVH_FACES mode). Attributes are created via BKE_sculpt_attribute_ensure.
Attributes can be explicit CustomData attributes or simple array-based pseudo-attributes (this is useful for PBVH_GRIDS and PBVH_BMESH).
SculptAttributePointers
There is a structure in SculptSession for convenience attribute pointers, ss->attrs. Standard attributes should assign these; the attribute API will automatically clear them when the associated attributes are released. For example, the automasking code stores its factor attribute layer in ss->attrs.automasking_factor.
Naming
Temporary attributes should use the SCULPT_ATTRIBUTE_NAME macro for naming, it takes an entry in SculptAttributePointers and builds a layer name.
SculptAttribute
Attributes are referenced by a special SculptAttribute structure, which holds
all the info needed to look up elements of an attribute at run time.
All of these structures live in a preallocated flat array in SculptSession, ss->temp_attributes. This is extremely important. Since any change to the CustomData layout can in principle invalidate every extant SculptAttribute, having them all in one block of memory whose location doesn't change allows us to update them transparently.
This makes for much simpler code and eliminates bugs. To see why this is tricky to get right, imagine we want to create three attributes in PBVH_BMESH mode and we provide our own SculptAttribute structs for the API to fill in. Each new layer will invalidate the CustomData block offsets in the prior one, leading to memory corruption.