IO: speed up large Alembic & USD imports by doing fewer collection syncs
ClosedPublic
Actions

Authored by Aras Pranckevicius (aras_p) on Jun 16 2022, 12:17 PM.

Details

Reviewers

Michael Kowalski (makowalski)
Bastien Montagne (mont29)

Commits

rB230f72347a99: IO: speed up large Alembic & USD imports by doing fewer collection syncs

Summary

Previous code was doing N collection syncs when importing N objects (essentially quadratic complexity in terms of object count). New code avoids all the intermediate syncs by using BKE_layer_collection_resync_forbid and BKE_layer_collection_resync_allow, and then does one BKE_main_collection_sync + BKE_main_collection_sync_remap for the whole operation. The things done on the importer objects that are dependent on the sync happening (marking them selected) are done in a separate loop after the sync.

The implementation uses the same "forbit sync" machinery as added & used in D11889, and already in-use by the OBJ importer.

Timings: importing Moana USD scene (480k objects) on Windows, VS2022 Release build, AMD Ryzen 5950X: 12344sec -> 10979sec (saves 22 minutes).

Diff Detail

Repository: rB Blender

Event Timeline

Aras Pranckevicius (aras_p) requested review of this revision.Jun 16 2022, 12:17 PM

Aras Pranckevicius (aras_p) created this revision.

Aras Pranckevicius (aras_p) mentioned this in D15222: USD: speed up large USD imports by not rebuilding material name map for each object.Jun 17 2022, 9:47 AM

I think one of the options for loading USD files is to load it in the background - I'm not certain of the detailed implementation implications of it, but if it's running on a separate thread would forbidding the collection sync interfere with someone trying to continue work while the file loads?

In D15215#410666, @Sonny Campbell (SonnyCampbell_Unity) wrote:

I think one of the options for loading USD files is to load it in the background - I'm not certain of the detailed implementation implications of it, but if it's running on a separate thread would forbidding the collection sync interfere with someone trying to continue work while the file loads?

The "background" case only means that there's an import progress bar in the status bar. The actual user interface & any interactions are still locked; you can't "do" anything while the import happens.

Even ignoring all that, all this change happens inside import_endjob of USD import, which is on the main thread.

A bit sad to have to see this hack (BKE_layer_collection_resync_forbid) used beyond specific internal Blender code, but I guess for now we have to go with it.

So LGTM.

This revision is now accepted and ready to land.Jun 17 2022, 5:01 PM

Closed by commit rB230f72347a99: IO: speed up large Alembic & USD imports by doing fewer collection syncs (authored by Aras Pranckevicius (aras_p)). · Explain WhyJun 17 2022, 9:35 PM

This revision was automatically updated to reflect the committed changes.

Aras Pranckevicius (aras_p) added a commit: rB230f72347a99: IO: speed up large Alembic & USD imports by doing fewer collection syncs.