Page MenuHome

Fix T96294: Crash and error with shape key normal calculation
ClosedPublic

Authored by Hans Goudey (HooglyBoogly) on Mar 11 2022, 8:44 PM.

Details

Summary

A mistake in the mesh normal refactor caused the wrong mesh to
be used when calculating normals with a shape key's deformation.

This commit fixes the normal calculation by using the correct mesh,
with just adjusted vertex positions, and calculating the results
directly into the result arrays when possible. This completely avoids
the need to make a local copy of the mesh, which makes sense,
since the only thing that changes is the vertex positions.

Diff Detail

Repository
rB Blender

Event Timeline

Hans Goudey (HooglyBoogly) retitled this revision from Fix T96294: Crash and error with shape key normal calculation (Option 1) to Fix T96294: Crash and error with shape key normal calculation.Mar 11 2022, 8:54 PM
  • Fix: Use correct mesh variable
Campbell Barton (campbellbarton) requested changes to this revision.EditedMar 17 2022, 1:42 AM

in principle I think it should be possible to do a shallow copy on a mesh to swap out spesific custom-data layers (vertex coordinates in this case).

It may be that there needs to be an API call to create & free shallow copies to prevent mistakes and avoid having low level custom-data handling inline as it is currently.

BKE_mesh_copy_for_eval is reasonably expensive, copying animation-data, ID-properties, materials, deform-groups, initializing thread mutexes, adding a session UUID ... as well as BKE_library_foreach_ID_link() that walks over materials, animation-data, id-properties etc, although not recursively.

Suggest either of the following:

  • Support a shallow copy API for mesh geometry evaluation which provides a convenient way to make a short lived shallow copy to be used & freed immediately afterwards, allowing custom-data layers to be replaced.
  • As this function happens only to be used from RNA, commit this patch with a warning this function should not be used during runtime/animation updates/evaluation and use is limited to the RNA-API, also an explanation for why BKE_mesh_copy_for_eval is used with a reference to the crash. In the future storing vertices as a float array means coords from either the key-block or the mesh could be used interchangeably.
This revision now requires changes to proceed.Mar 17 2022, 1:42 AM
Hans Goudey (HooglyBoogly) edited the summary of this revision. (Show Details)
Hans Goudey (HooglyBoogly) set the repository for this revision to rB Blender.
  • Calculate normals directly into result arrays with the input mesh

With a bit of logic, we can completely remove copying a mesh, even a local copy on the stack.
I think this is preferable to using a whole new API to create a local copy of the mesh, since
in the end we're really just dealing with a few arrays here.

@Campbell Barton (campbellbarton) Let me know what you think. I think this might be the best solution, a good between my two original patches.

This revision is now accepted and ready to land.Mar 22 2022, 8:21 AM