Page MenuHome

Fix T88756: crash when baking with autosmooth
ClosedPublic

Authored by Kévin Dietrich (kevindietrich) on Jun 25 2021, 1:04 PM.

Details

Summary

When baking some data, we create a new Mesh with edits and modifiers applied.
However, in some cases (e.g. when there is no modifier), the returned Mesh is
actually referencing the original one and its data layers. When autosmooth is
enabled we also split the Mesh. However, since the new Mesh is referencing the
original one, although BKE_mesh_split_faces is creating new vertices and edges,
the reallocation of the custom data layers is preempted because of the
reference, so adding the new vertices and edges overwrites valid data

To fix this we duplicate referenced layers before splitting the faces.

Diff Detail

Repository
rB Blender
Branch
fix_T88756 (branched from master)
Build Status
Buildable 15431
Build 15431: arc lint + arc unit

Event Timeline

Kévin Dietrich (kevindietrich) requested review of this revision.Jun 25 2021, 1:04 PM
Kévin Dietrich (kevindietrich) created this revision.
This revision is now accepted and ready to land.Jun 25 2021, 2:10 PM

Actually, can we do this only when we know we need new vertices, and only for vertices and edges right before CustomData_realloc is called?

I guess now it could use additional memory when not needed.

Actually, can we do this only when we know we need new vertices, and only for vertices and edges right before CustomData_realloc is called?

I initially tried to only create new layers only when necessary, but then I got a crash when the split function is called from the Cycles side, since loop indices are already modified on the original data. (This crash happens at the beginning of the function, when computing normals, before trying to split vertices).

This will require to change the logic in split_faces_prepare_new_verts to not update the edges and loops indices for new vertices, or do the layers duplication there when we know for sure that vertices will be added.