This task is here to explain current design behind lybrary overrides in #bf_blender_2.8, and to keep track of known limitations and TODOs.
Reminder: “library override” only allows overriding linked data. Since those do not change, we can evaluate library overrides once (when loading .blend, or re-loading a library). This could also be named 'partially localized data-blocks'. On the other hand, “dynamic override” would allow overriding local data, and would be handled by the depsgraph evaluation to generate final data.
This is not finalized project, but most (all) core concepts, API and code are expected to be final. Lots remain to be done mostly in adding overridable status to many more properties, and… stress-test all kind of complex corner cases!
Right now, only a few properties are overridable, among which object's and bone’s loc/rot/scale, you can test the code by:
- linking an object;
- make an override of it (through operator, type 'override' in space-bar search menu);
- change e.g. location, rotation and/or scale of the override;
- save file;
- open file again, and see that changed location/rotation/scale were kept.
You may also note that non-overridable properties remain grayed-out in UI.
Finally, at least basic support of animation override is done and working (i.e. being able to add, or replace, animation of the local overriding ID).
TODO's for 2.81
============
Those are known tasks that should be addressed before we go into 2.81 beta phase (i.e. we create the release branch).
[x] {icon circle color=red} {T54652}.
TODO's for 2.82:
-------------------
[] {icon circle color=red} Define much more RNA properties as overridable (this is easy-monkey work mostly).
[] {icon circle color=yellow} {T73154}
[] {icon circle color=yellow} Add/extend operator to override chains of dependencies (you select an ID to override, and it automatically also overrides all/some of its linked users up to the first local data-block) - mostly for Outliner?
[] {icon circle color=yellow} Add an operator to fully revert an override to its linked data-block (it's basically ID remapping to linked data + deletion of the override).
[] {icon circle color=yellow} Add an operator to fully reset an override to its linked data-block state. //Warning: likely more complicated than it looks, as this should not break the overridden pointers to other IDs in the same 'overriding family'.//
[] {icon circle color=yellow} Add an operator to convert from existing proxies to new lib overrides.
TODO's for 2.83:
-------------------
[] {icon circle color=red} Define much more RNA properties as overridable (this is easy-monkey work mostly).
[] {icon circle color=red} {T73156}
[] {icon circle color=yellow} Solve/define a proper way to handle materials overrides?
[] {icon circle color=green} Add a setting similar to auto-keyframe to automatically generate an override when you edit a property? //NOTE: this is likely very complex to get right, as by default linked IDs are fully non-editable...//
TODO's for later:
--------------------
[] {icon circle color=yellow} {T72629}
[] {icon circle color=green} {T54762}, partially addressed already, but it should be possible to improve things further still.
Technical Details
============
Here is a summary of how library overrides work.
# **Overrides are handled at RNA level** That means we use RNA paths to control which property are overridden (much like animation in fact).
# **Overrides are 'rooted' at ID level** That means that a given ID stores all its overriding, including those in “sub-data” structures.
# Each overridden, local ID keeps a pointer to its (linked) reference ID.
# Overrides behave exactly like any other local data-blocks, for 99% of Blender code. Only places that need to be aware of their status are:
## UI (through RNA, to give feedback to the user).
## Load and save code.
## Some way to hook the auto-override creation when someone changes an override's property (done as part of undo step creation currently).
### RNA changes.
The way RNA properties are compared was heavily rewritten to allow much more complex behavior. Aside from override-specific topics, this brings real comparison (and not only-equality one), and more importantly, the ability to have a custom callback for custom comparison process.
### Override Creation
This is merely making a local copy of the linked datablock, and generate an empty override structure for it.
[X] We need some advanced and complex behavior to be possible here (especially thinking about rigged chars, where you need to override armature and mesh objects, and relink the relations between them…).
[X] Hook up override creation into UI (add menu entries, etc.). Easy, to be done once all known issues are fixed.
### Auto-generating override operations
User can define/control overriding of properties by hand, but we also generate them automatically, by making some RNA-based comparison between reference and local IDs. RNA comparison code has been extensively enhanced to support that. Auto-generation is currently performed each time an undo step is generated.
[X] Support for some collections (modifiers or constraints e.g.).
[X] Refined handling of pointers. Pointers can be to other IDs (then we only want to compare pointer itself), or to some sub-data we **own** (in which case, we want to go further down the path), or to some other we **do not own** (we only want to compare pointer itself here too).
[ ] Try to improve performances (T54762) (for big production characters, this can add several hundreds of milliseconds to every undo step... we can live with it, but this could be improved). //Partially addressed already, but it should be possible to improve things further still.//
### Override at .blend saving
We save local override 'as is', along with its override operations list. In case some operations are differential (add, sub, multiply), we compute and store the other operand in another datablock (storage one). This simplifies/sanitize write code (and most importantly, avoid having to temporarily modify real ID data during save process!), and also allows to get the 'overridden state' of the ID even when opening in an older Blender unaware of overriding process.
[ ] Right now we make full usual copy of the local overriding ID, which is not acceptable for huge datablocks (thinking especially about mesh data here!), we'll need a way to trim useless data (probably using a new copy flag). Not critical though, since Mesh (and other data) IDs are not overridden currently. Note however that removing actual geometry data will make file fully broken in case lib data is lost, or if loaded in an older blender version... Not sure how critical this is in either way.
### Override at .blend loading
Once we have loaded the whole .blend file and all its libraries, we go over whole Main database and apply overrides. To do that without having to recreate, and remap, and rename whole IDs, we edit a copy of linked (reference) ID, and then swap its content with local one.
[ ] Would be nice to investigate improving a bit code efficiency here, if possible. E.g. we could make out-of-main copy, but this would then probably break ID refcounting… Rather low priority anyway, current code does not seem to be a serious bottleneck.
Known TODOs, Issues, etc.
====================
**This is not feature complete!** By far… Only basic operations are implemented, several areas remain fuzzy, etc. However, core concepts and code should be mature enough to be merged in blender2.8, where ongoing development can go on. Also, current state is enough and needed for Asset Engine project.
### Code Working on DNA
Mainly operators, but can be also a few buttons using low-level access to data…
This is a bit of a hairy issue, since overriding data-blocks are essentially local ones, any operator can go playing on their data with any knowledge about what they should be allowed to do!
A first half of the solution is done, which is adding a post-operation check to try to restore data from reference when needed (added as part of the auto-override generation in undo step).
This does not solve everything though, we cannot undo all changes from RNA. So second part of the solution will unfortunately be to add checks to operators. We //probably// can limit this to a general check over a datablock (at least for a start), to forbid some operators to work on overriding ones completely.
But reaching a full and perfect control on the situation here is likely to be very difficult and long, if possible at all… So question is, to what point can we accept to have some changed data in local overrides that shall not be different from their reference? Knowing that a save & reload of the .blend file (or a manual 'resync' from linked references) will restore things to their correct status?
### Override Templates & Locked Override
Those features are partially implemented, but require more work to be usable by users. They are not high-priority currently, and are not being actively worked on.
[ ] Differential overrides (add/subtract, or multiplication factor, also binary operation for flags, etc.). Mostly implemented, but need some UI/UX design work.
[ ] Override templates: those are overrides defined in the library file. They give a default override “pattern” when creating a new override in final files (e.g. for a rig, it will allow to restrict which bones are editable in the overrides, and which will remain 'locked' for the end user - at least from UI, PyRNA API will allow to do everything of course).
[ ] Locked override: by defining a NO-OP (no operation) override operation for a given property, you can effectively prevent any overriding of that property (ensure it will always remain at same value as the linked overridden data-block).
### IDProperties
T54652
[X] Even though they were taken into account to some extent into new code, those are not supported currently. This needs to be addressed, but don’t think it's super-high priority for now.
### Non-trivial cases - Rigged chars
While a lot of tests were done, and a lot of issues fixed, current code should not yet be considered fully ready to replace proxies.
[X] Support complex rigs with lots of constraints, drivers and other nifty features.
[X] Support production chars like Autumn (a.k.a uber-complex rigs, with sub-rigs etc.).
### Non-trivial cases - Materials
[ ] The way materials are linked to object or obdata make them especially tricky to handle in override case.
### Non-trivial cases - Nodes
[ ] Not yet checked, but fairly sure that the way nodetrees can be real or fake IDs (root trees of materials/compositing/etc.) will also be an issue…