Page MenuHome

Fix T100203: Freeze using `override_hierarchy_create` with Object level property with custom getter function
AbandonedPublic

Authored by Bastien Montagne (mont29) on Aug 9 2022, 3:51 PM.

Details

Summary

It is not possible to process in threads IDTypes that have py-defined
RNA properties with custon callbacks, since those are executed in py,
which is only possible in the main thread.

So introduce a way to check whether a given ID type has py-defined
RNA properties with cutom callbacks, in some specific context (e.g. in
liboverride case, if a property is explicitely excluded from comparison,
we can still process those IDs even if it has python callbacks).

Diff Detail

Repository
rB Blender
Branch
tmp (branched from master)
Build Status
Buildable 23307
Build 23307: arc lint + arc unit

Event Timeline

Bastien Montagne (mont29) requested review of this revision.Aug 9 2022, 3:51 PM
Bastien Montagne (mont29) created this revision.

Executing Python code is not limited to the main thread, we do it in other threads for render engines and python drivers for example.

For me the deadlock looks like this: P3135. The problem is that the thread is trying to execute Python code will the main thread is still holding the global interpreter lock.

For similar recursive and threaded Python calls we use BPy_BEGIN_ALLOW_THREADS in the RNA API. Like this: P3136. But I haven't checked if it's the only place that would need this.

@Brecht Van Lommel (brecht) but the issue here is that there could be several IDs of the same type processed concurrently, and there is nothing in a custom py-defined getter/setter that prevents them to access data from another ID, or any other kind of stuff breaking thread safety?

C-defined 'official' RNA code is supposed to know it should not access data outside of the owner ID, but python code can do whatever it wants :|

This patch only executes the affected datablocks on the main thread, while other datablocks are still threaded. If a callback on the affected datablock reads from another type of datablock, it's also not threadsafe. So you'd need to make the whole thing single threaded, which seems like a big cost.

On the other hand, RNA get/set are already assumed thread safe by the depsgraph (for example for animation curve evaluation, not even Python drivers). And there is some thread safety between multiple threads executing Python, as calls into the Blender API will be protected by the GIL.

So while you can break things when get/set access other datablocks, I'd rather we add a note to the API docs that get/sets should only access data within the datablock.

So while you can break things when get/set access other datablocks, I'd rather we add a note to the API docs that get/sets should only access data within the datablock.

At least for now we do need to add such a note. Otherwise if one uses such custom getter and animates the custom property the DEG will not handle it properly since there is no relation in the DEG to ensure order of evaluation.

For the initial report, I'm not sure why the rna_access_threadsafe is limited to armatures only. Seems that BPy_BEGIN_ALLOW_THREADS/BPy_END_ALLOW_THREADS will be more generic solution.
Or, maybe even extra work is needed to fully resolve custom getters/setters since those might be used by animation system from multiple threads.

@Sergey Sharybin (sergey) rna_access_threadsafe callback is only defined for armature and objects because those have custom sub-data (bones....) that can also have py-defined properties. afaik all other IDs can only get py-defined properties in the ID type itself, which is handled generically by BKE_idtype_idcode_is_rna_access_thread_safe.

@Brecht Van Lommel (brecht) aaaaah fair enough... then yes, think it's simpler for now to use BPy_BEGIN_ALLOW_THREADS/BPy_END_ALLOW_THREADS here as well. Will also update the py api doc about that thread-safety point.