The goal of this project is to implement a new Point Cache system based on the [[ https://code.google.com/p/alembic/ | Alembic library ]] and unify the various caching methods used in Blender.
This is a prerequisite for further work on particle systems. Having a working Alembic exporter/importer will also greatly benefit integration of Blender into larger pipelines.
=== Current Issues ===
* Point cache is used for most physics simulations, but does not include mesh caching. For this there is a separate [[ http://wiki.blender.org/index.php/Doc:2.6/Manual/Modifiers/Modify/Mesh_Cache | Mesh Cache modifier ]], using mdd or pc2 file formats, with its own limiations. Using a unified point cache system would help scene management.
* Alembic, unlike the current point cache binary format, supports arbitrary data attributes. This is very important for flexible simulations and user-defined data in particles and mesh data (vertex groups, [[ http://wiki.blender.org/index.php/Dev:Source/Architecture/Custom_Element_Data | custom data layers ]], etc.).
* Point cache is limited to fixed-size data sets. This is sufficient for many simulations which only deform geometry (cloth, hair), but for particles and constructive modifier caching variable point data sets are important.
* Compression in the point cache file format leaves a lot to be desired. Alembic is specifically designed to have a small memory/disk space footprint and make good use of threading for fast export/import.
* Point cache binary format is unique to Blender. While it is fairly simple to interpret, it does not facilitate import/export and integration into larger pipeline workflows. Alembic on the other hand is a commonly used tool and can leverage a large developer community.
* Settings for caches are object-centric, each simulation writes its own separate cache files. Baking all caches is simply a loop over all objects, which is not very efficient. Having scene-level settings and operators would make Alembic export/import easier.
=== Proposal ===
# Keep basic cache settings on Scene level:
- "Disk Cache" vs. "Memory Cache": This could be reframed as having cache file output by default and use ''packing'' in .blend files like we do with images.
- Output directory setting for cache files (with nice defaults)
- Frame ranges
- Automatic cache recording settings (see below)
# Optional overrides on object/sim level
If deviating settings are needed for individual simulation caches. By default it should not be necessary to tweak each cache individually.
# **Question**: How do we handle automatic caching?
Current approach is to automatically record point cache when stepping through frames, unless the cache is explicitly baked.
This breaks easily when jumping over frames and or scrubbing in the timeline, because simulations need a contiguous ordered frame range for correct results.
Does this work well enough to justify keeping it? Are there alternatives to this behavior?
One suggestion was to make simulations independent from the animation playback by default, but this makes it difficult to sync simulation results to keyframe-animated objects and to record valid caches on-the-fly.
# **Question**: Do we really need multiple caches for each object? This feature adds a list of point caches for every object, only the active cache is used, the others are basically a version control system. Could this be moved to the scene level?
==== Directory and File Names ====
How should directories and file naming work?
Current naming scheme works per object and uses a combination of output directory + filename:
- Directory is chosen in this order:
# explicit path if specified by the user
# or constructed as `//blendcache_<blend file name>/` (if .blend is saved and thus relative paths work)
# or using a temp folder like `<tmp folder>/blendcache/`
- File name is generated as `<basename>_<frame number>_<cache index>.bphys`
- `<basename>` can be specified by the user or otherwise the ID datablock name (usually Object) is used
- `<cache index>` is generated automatically by default (-1), but needed if multiple caches exist in the same ID block.
==== Frames, Times and Sample Indices ====
There are 3 different modes of interpreting "time" that need to be managed in a clean way:
* Blender usually defines data in terms of **frame**.
Any notion of "time" is secondary, defined by the frames-per-second (fps) settings in the scene and additional start/end frames.
Some simulations also add their own interpretation of time on top of this, by using time scale factors (e.g. [[ http://wiki.blender.org/index.php/Doc:2.6/Manual/Physics/Smoke/Domain#Generic_options | smoke sim time scale options ]]), although these factors are mostly for the internal simulation time step and don't have much consequence for mapping sample times.
* Alembic samples can be accessed
- directly by an **index** (0..N)
- or by using a **time** value which is mapped to a sample index based on a time sampling (see [[ https://code.google.com/p/alembic/wiki/AlembicPoint9UsersGuide#Time_Sampling | Time Sampling overview ]], [[ https://code.google.com/p/alembic/source/browse/lib/Alembic/Abc/ISampleSelector.h#47 | ISampleSelector class ]])
While it would be possible to largely ignore time sampling for Blender cache files and simply read each cached sample as a frame, it is probably more reliable and flexible to calculate time values as defined by the scene render range and fps settings. This would allow using externally generated Alembic files which use a different sampling rate than the current scene and map them into the scenes frame range.
Ultimately it could be very useful to integrate cached data into the NLA editor as a track. At that point the direct sample<->frame relationship that might still exist for cached data from the same scene is useless anyway.
**Proposed Changes**
* Output directory is by default specified on scene level, using the same rules as before.
* Alembic cache files don't store files for each individual frame, instead one file for the whole cached frame range is used
* 2 possible behaviors:
- Each cached simulation/modifier creates its own file in the cache folder
- Combine caches in a single scene cache file.
This is more in line with standard Alembic use, but makes it more difficult to cache objects separately
* In both cases identifier names for cache data are a problem.
This is already the case with current point caches: Renaming an object will sever the mapping to file names, leading to unused files on the disk and making re-bake necessary.
=== Implementation ===
{F29943}
# Scene stores global settings for caching
# Simulations and modifiers can have own settings if needed
# Cache data is separate from other DNA by default, but can be packed into blend files like images
# Point Cache is a thin API, using Alembic as a backend
# Different schema implementations (writer/reader) are created for the various DNA types and to handle interpolation (comparable to current [[ http://developer.blender.org/diffusion/B/browse/master/source/blender/blenkernel/BKE_pointcache.h;885354daec2fa8181aeb27b637c80e4ee4c247df$122 | PTCacheID ]])
# Scene writer/reader ties everything together for exporting the scene as a whole, possibly using object hierarchy (flattening scene structure for export would also be possible)
=== Notes ===
* Build instructions for Alembic and the development branch:
http://wiki.blender.org/index.php/User:Phonybone/PointCache/BuildNotes
* Unsorted list of thoughts on Alembic implementation details:
http://wiki.blender.org/index.php/User:Phonybone/PointCache/InitialThoughts
* Original [[ https://wiki.gnome.org/Apps/Dia | dia ]] file for the diagram: {F29958}