Details

Reviewers

Commits

rB6f96dfabe5cb: Simulations: initial simulation state and cache

Summary

I'd like to use this "thread" to decide how we want to store particle attributes
in the .blend file. The approach I implemented in the patch below is just one of
many possibilities. However, I think it helps to have something that "works" as
a basis for the discussion. I'm fine with trashing all the code below. It's very
work-in-progress anyway.

Some aspects to consider

Should we use CustomData or the new attribute system from T76659?
Should we use the existing PointCache system?
Should the cache be stored on the original data block or on the COW simulation?

Some Context

A Simulation is a container for a "simulation description" (node tree) and
the cached simulation result. A simulation data block can contain multiple
particle simulations and later also other kinds of simulations. Different
types of simulations require different storage types. We can't really make
a lot of assumptions about the data format imo.
Furthermore, it would be good to be able to have the entire cache in the .blend
file or on disk.

The cached simulation data is accessed by a Simulation modifier on e.g. point cloud objects.
This modifier copies the simulation data into the object data for further editing and rendering.

Current Implementation

A Simulation has an arbitrary number of SimulationCaches.
SimulationCache is a "base class" for different kinds of caches.
For now, there is only ParticleSimulationCache.
There could also be FluidSimulationCache, etc.
There can be multiple caches with same type.
Every cache has a name that will be used as identifier by the simulation modifier.
A ParticleSimulationCache has a ParticleSimulationFrameCache for every cached frame.
A ParticleSimulationFrameCache stores how many particles there are and the attributes.

Everything referenced by ParticleSimulationCache could probably be replaced by some
existing functionality like CustomData or PointCache. I'm not sure if there is a
big benefit in using those. Reinventing the wheel does not feel too costly, yet.

Some feedback on the implementation described above is welcome.
Alternatively, I'd like to hear thoughts on a completely different implementation as well.

Diff Detail

Repository

rB Blender

Branch

simulation-cache (branched from master)

Build Status

Buildable 8199
Build 8199: arc lint + arc unit

Event Timeline

Jacques Lucke (JacquesLucke) requested review of this revision.May 25 2020, 5:40 PM

Jacques Lucke (JacquesLucke) created this revision.

Harbormaster completed remote builds in B8198: Diff 25076.May 25 2020, 5:40 PM

Jacques Lucke (JacquesLucke) edited the summary of this revision. (Show Details)May 25 2020, 5:40 PM

Update patch.

Harbormaster completed remote builds in B8199: Diff 25077.May 25 2020, 5:43 PM

Jacques Lucke (JacquesLucke) edited the summary of this revision. (Show Details)May 25 2020, 5:45 PM

Jacques Lucke (JacquesLucke) edited the summary of this revision. (Show Details)

Jacques Lucke (JacquesLucke) retitled this revision from Simulations: DNA cache structure [WIP] to Simulations: DNA cache structure [Discussion].

Jacques Lucke (JacquesLucke) added a subscriber: Brecht Van Lommel (brecht).

• Ben Sholly (bobbrown) edited the summary of this revision. (Show Details)May 25 2020, 6:53 PM

Brecht Van Lommel (brecht) edited the summary of this revision. (Show Details)May 25 2020, 6:55 PM

Should we use CustomData or the new attribute system from T76659?

I consider CustomData and a new attribute system to be the same thing. The same underlying data structures that are getting refactored / extended. If the simulation cache should use the same data structures is another matter.

I think the cache system doesn't need to use CustomData and CustomDataLayer data structures. The cache may be on disk or in memory, and the data might be compressed in some way (e.g. delta compression between frames or LZ4 or whatever). The cache data structures can have some flexibility for that kind of thing while CustomDataLayer wouldn't.

Should we use the existing PointCache system?

I think this is actually the best way to go. PointCache has room for improvement, but I also can't think good reasons against using it. To me it actually seems like a decent fit already.

Getting a cache system right is not easy, and much of the complexity that is in the current implementation is actually there for a good reason. I think we have a better chance of getting a good results by improving PointCache than starting from scratch.

Should the cache be stored on the original data block or on the COW simulation?

I think the same mechanism as PointCache should be used. Basically it's owned by the original datablock and not duplicated by COW, there is only a single cache that is reference. COW datablocks in the active dependency graph can write to it, for other dependency graphs it's read-only.

Erick Tukuniata (erickblender) awarded a token.May 27 2020, 10:35 PM

I think the same mechanism as PointCache should be used. Basically it's owned by the original datablock and not duplicated by COW, there is only a single cache that is reference. COW datablocks in the active dependency graph can write to it, for other dependency graphs it's read-only.

Thanks, that was useful.

Should we use the existing PointCache system?

I think this is actually the best way to go.

Good, I'll try to implement the same basic functionality as this patch using PointCache. I will probably reference the point cache from ParticleSimulationCache somehow.

Do you think the SimulationCache abstraction is fine? Since we only have particles for now, we could also get rid of this abstraction layer and just have a list of ParticleSimulationCaches.

Thinking about this, I'm not sure it's helpful to have multiple PointCache instances per Simulation datablock? For managing file paths, frame ranges, baking and freeing bakes, seeing which frames are cached or baked, baking, ..., it seems easier for users if there is just a single cache per animation. If users somehow have to manage multiple I think that will be quite cumbersome.

Different parts of the same simulation might be stored in different files or have different frame ranges where they actually need to cache anything. But perhaps that can be hidden from the user?

Regarding SimulationCache and ParticleSimulationCache, I don't mind having the abstraction there even if we don't use it yet. But it's not clear to me what exactly will be stored in there if PointCache is used.

Philipp Oeser (lichtwerk) added a subscriber: Philipp Oeser (lichtwerk).May 28 2020, 7:54 PM

This might be a bit off-topic [not sure if there is a better place to ask?], I was just wondering about the future of Particle Editmode [future Cache Editmode?].
I assume that if PointCache is kept, most of the functionality around PTCacheEdit will as well?

Of course some hair-specific stuff probably needs to be pulled out, but generally we would want to continue supporting editing of particle/simulation cache paths, right?
Atm. this is pretty much broken/unsupported/paths not drawn etc. for everything [other than editing hair itself -- but editing hair dynamics is also broken]: cloth, softbodies, emitter particles...

Just asking, if PTCacheEdit and friends are generally supposed to be kept, it might make sense looking into fixing some of the broken functionality.

I think point cache editing mostly needs to be rewritten. For meshes it should go though the same code paths as shape key editing and work in edit and sculpt mode, rather than a particle edit mode.

I would only look into that once we are further along with replacing the old particle system.

Philipp Oeser (lichtwerk) mentioned this in T77114: No way to edit baked cloth simulation with particle edit mode..May 29 2020, 11:30 AM

I tried a different way to store the data. It's now using CustomData for
the current particle state and PointCache for caching over time. I also
integrated it with the timeline. This approach seems to work quite well so far.
I'll probably have to do some refactorings in PointCache that I'll look into next.

I agree that from a user point of view there should usually just be a single cache.
I'm not sure how well that translates to having a single PointCache though. If a
simulation contains two particle systems, those would need two separate PointCaches
for now.

Harbormaster completed remote builds in B8330: Diff 25400.Jun 2 2020, 5:33 PM

This seems a good start, we can worry about having a single PointCache later.

This revision is now accepted and ready to land.Jun 2 2020, 9:03 PM

I just wanted to commit this when I noticed that the COW handling seems to be a bit wrong here. Currently, the cache is only stored on simulation_orig, which is what we want. However, the "current state", i.e. the custom data, also only exists on simulation_orig. That means that a simulation cannot be evaluated in different frames at the same time, even though there are multiple COW copies. I see two possible solutions:

Have a ListBase states and a ListBase caches in the Simulation struct. On simulation_orig, every SimulationState has exactly one SimulationCache and vice versa. On simulation_cow, the states list is copied, but the caches list is empty.
Keep the data structures as they are right now. However, when a cow copy is created, only attributes is copied; point_cache is set to nullptr. Maybe SimulationState should have a backlink to its counterpart on simulation_orig to lookup the cache (similar to orig_modifier_data). Without this backlink it would have to traverse the states list on the simulation_orig to find the matching state. BKE_simulation_data_update then modifies attributes on simulation_cow and stores the cache on simulation_orig.

I prefer option 2 currently.

Option 2 seems fine.

Ok, I implemented that second option.

In general the code in BKE_simulation_data_update works like so now:

if (original state has cache for current frame):
    load cache into cow state
elif (depsgraph is active):
    if (is first frame):
        initialize orig state
        store orig state in cache
        copy orig state to cow state
    elif (orig state contains previous frame):
        simulate rom previous to current frame
        store orig state in cache
        copy orig state to cow state

There is one thing that confuses me a bit. When I first open a file with
a partially cached simulation, the modifier is evaluated on file load, but
the simulation is not updated. The updateDepsgraph function of the modifier
is adding the relationship with the simulation before the modifier is evaluated.
Only when I change the frame after the file is loaded, the simulation is updated.
Is there some optimization that skips some data blocks in the initial depsgraph
evaluation when a new file is loaded? @Sergey Sharybin (sergey)

Harbormaster completed remote builds in B8370: Diff 25481.Jun 4 2020, 1:13 PM

That is a bit vague question. There isn't a function which takes care of an entire datablock evaluation, updates happen in a granular manner.
There is some code in deg_graph_on_visible_update which tags different components of datablock to be evaluated. This is run any time new dependency graph is created from scratch, or when visible layer changes.

You can use simple scene and --debug-depsgraph to have more clues about what's going on.

update read/write code to use new api
minor fix when copying state from original to cow simulation
tag simulation for update, when it is used by the simulation modifier

I think this should be good enough for master now. Will do further
development in separate patches.

@Sergey Sharybin (sergey) thanks. In the end it came down to tagging the simulation when
the modifier starts using it. It's working as I'd expect now.

Another quick video showing what this patch does.

Harbormaster completed remote builds in B8428: Diff 25632.Jun 8 2020, 12:40 PM

Brecht Van Lommel (brecht) accepted this revision.Jun 8 2020, 3:31 PM

Closed by commit rB6f96dfabe5cb: Simulations: initial simulation state and cache (authored by Jacques Lucke (JacquesLucke)). · Explain WhyJun 8 2020, 3:50 PM

This revision was automatically updated to reflect the committed changes.

Jacques Lucke (JacquesLucke) added a commit: rB6f96dfabe5cb: Simulations: initial simulation state and cache.

Simulations: DNA cache structure [Discussion]
ClosedPublic
Actions

Details

Some aspects to consider

Some Context

Current Implementation

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 25077

source/blender/blenkernel/BKE_simulation.h

source/blender/blenkernel/intern/pointcloud.c

source/blender/blenkernel/intern/simulation.cc

source/blender/blenloader/intern/readfile.c

source/blender/blenloader/intern/writefile.c

source/blender/makesdna/DNA_simulation_types.h

source/blender/modifiers/intern/MOD_simulation.cc

Simulations: DNA cache structure [Discussion]ClosedPublicActions

Details

Some aspects to consider

Some Context

Current Implementation

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 25077

source/blender/blenkernel/BKE_simulation.h

source/blender/blenkernel/intern/pointcloud.c

source/blender/blenkernel/intern/simulation.cc

source/blender/blenloader/intern/readfile.c

source/blender/blenloader/intern/writefile.c

source/blender/makesdna/DNA_simulation_types.h

source/blender/modifiers/intern/MOD_simulation.cc

Simulations: DNA cache structure [Discussion]
ClosedPublic
Actions

Revision Contents
Changeset List