Fix assertion failure when converting BMesh to Mesh with shape keys
ClosedPublic
Actions

Authored by Sybren A. Stüvel (sybren) on May 22 2020, 1:42 PM.

Details

Reviewers

Campbell Barton (campbellbarton)
Sergey Sharybin (sergey)
Brecht Van Lommel (brecht)

Commits

rB9b9f84b317fe: Fix crash when converting BMesh to Mesh with shape keys

Summary

The BM_mesh_bm_to_me() function copies shape keys from the BMesh to the Mesh. However, it tries to copy the same number of shape keys as are defined on the target mesh. Since the target mesh does not necessarily have the same number of shape keys as the BMesh, this would crash if the target Mesh has more.

Found while performing some tests for D7785: Fix T65148: Can't use shape key properties in driver when modifiers generate a new mesh.

To reproduce the issue:

Open the attached blend file in a debug build of Blender.
Run the code in the text editor.
Assertion failure:

blender: …/blender/source/blender/bmesh/intern/bmesh_mesh_conv.c:941: BM_mesh_bm_to_me: Assertion `cd_shape_offset != -1' failed.

D7818-example.blend146 KBDownload

Note that even with this fix, the remaining shapekeys on the mesh are invalid. It might be desirable to delete the shapekeys before or after the operation, but I feel that this is out of the scope of the BM_mesh_bm_to_me() function.

Diff Detail

Repository: rB Blender

Event Timeline

Sybren A. Stüvel (sybren) created this revision.May 22 2020, 1:42 PM

Harbormaster completed remote builds in B8164: Diff 25007.May 22 2020, 1:42 PM

Sybren A. Stüvel (sybren) requested review of this revision.May 22 2020, 1:42 PM

Sybren A. Stüvel (sybren) retitled this revision from Fix crash when converting BMesh to Mesh with shape keys to Fix assertion failure when converting BMesh to Mesh with shape keys.May 22 2020, 1:45 PM

Sybren A. Stüvel (sybren) edited the summary of this revision. (Show Details)

Sybren A. Stüvel (sybren) added projects: BF Blender (2.83), Modeling.

Sybren A. Stüvel (sybren) edited the summary of this revision. (Show Details)May 22 2020, 1:55 PM

Brecht Van Lommel (brecht) requested changes to this revision.May 22 2020, 2:24 PM

Brecht Van Lommel (brecht) added a subscriber: Brecht Van Lommel (brecht).

Brecht Van Lommel (brecht) added inline comments.

source/blender/bmesh/intern/bmesh_mesh_conv.c
898	Use `continue` instead.

This revision now requires changes to proceed.May 22 2020, 2:24 PM

Sybren A. Stüvel (sybren) mentioned this in D7785: Fix T65148: Can't use shape key properties in driver when modifiers generate a new mesh.May 22 2020, 3:28 PM

Sybren A. Stüvel (sybren) added inline comments.May 22 2020, 5:35 PM

source/blender/bmesh/intern/bmesh_mesh_conv.c
898	Why? Is it possible for a mesh to not have `CD_SHAPEKEY` with index `j` but to do have one with index `j+1`? Otherwise I don't see anything in this for-loop that should be called for every `CD_SHAPEKEY` layer in the target mesh.

Brecht Van Lommel (brecht) added inline comments.May 22 2020, 5:49 PM

source/blender/bmesh/intern/bmesh_mesh_conv.c
898	`j` is computed by `bm_to_mesh_shape_layer_index_from_kb`. It's not obvious that it is monotonically increasing. Even if it is, relying on such assumptions is fragile.