Change Details

The goal of this project is to implement a new Point Cache system based on the [[ https://code.google.com/p/alembic/ | Alembic library ]] and unify the various caching methods used in Blender. This is a prerequisite for further work on particle systems. Having a working Alembic exporter/importer will also greatly benefit integration of Blender into larger pipelines. === Current Issues === * Point cache is used for most physics simulations, but does not include mesh caching. For this there is a separate [[ http://wiki.blender.org/index.php/Doc:2.6/Manual/Modifiers/Modify/Mesh_Cache | Mesh Cache modifier ]], using mdd or pc2 file formats, with its own limiations. Using a unified point cache system would help scene management. * Alembic, unlike the current point cache binary format, supports arbitrary data attributes. This is very important for flexible simulations and user-defined data in particles and mesh data (vertex groups, [[ http://wiki.blender.org/index.php/Dev:Source/Architecture/Custom_Element_Data | custom data layers ]], etc.). * Point cache is limited to fixed-size data sets. This is sufficient for many simulations which only deform geometry (cloth, hair), but for particles and constructive modifier caching variable point data sets are important. * Compression in the point cache file format leaves a lot to be desired. Alembic is specifically designed to have a small memory/disk space footprint and make good use of threading for fast export/import. * Point cache binary format is unique to Blender. While it is fairly simple to interpret, it does not facilitate import/export and integration into larger pipeline workflows. Alembic on the other hand is a commonly used tool and can leverage a large developer community. * Settings for caches are object-centric, each simulation writes its own separate cache files. Baking all caches is simply a loop over all objects, which is not very efficient. Having scene-level settings and operators would make Alembic export/import easier. === Proposal === # Keep basic cache settings on Scene level: - "Disk Cache" vs. "Memory Cache": This could be reframed as having cache file output by default and use ''packing'' in .blend files like we do with images. - Output directory setting for cache files (with nice defaults) - Frame ranges - Automatic cache recording settings (see below) # Optional overrides on object/sim level If deviating settings are needed for individual simulation caches. By default it should not be necessary to tweak each cache individually. # **Question**: How do we handle automatic caching? Current approach is to automatically record point cache when stepping through frames, unless the cache is explicitly baked. This breaks easily when jumping over frames and or scrubbing in the timeline, because simulations need a contiguous ordered frame range for correct results. Does this work well enough to justify keeping it? Are there alternatives to this behavior? One suggestion was to make simulations independent from the animation playback by default, but this makes it difficult to sync simulation results to keyframe-animated objects and to record valid caches on-the-fly. # **Question**: Do we really need multiple caches for each object? This feature adds a list of point caches for every object, only the active cache is used, the others are basically a version control system. Could this be moved to the scene level? === Implementation === {F29943} # Scene stores global settings for caching # Simulations and modifiers can have own settings if needed # Cache data is separate from other DNA by default, but can be packed into blend files like images # Point Cache is a thin API, using Alembic as a backend # Different schema implementations (writer/reader) are created for the various DNA types and to handle interpolation (comparable to current [[ http://developer.blender.org/diffusion/B/browse/master/source/blender/blenkernel/BKE_pointcache.h;885354daec2fa8181aeb27b637c80e4ee4c247df$122 | PTCacheID ]]) # Scene writer/reader ties everything together for exporting the scene as a whole, possibly using object hierarchy (flattening scene structure for export would also be possible) === Notes === * Build instructions for Alembic and the development branch: http://wiki.blender.org/index.php/User:Phonybone/PointCache/BuildNotes * Unsorted list of thoughts on Alembic implementation details: http://wiki.blender.org/index.php/User:Phonybone/PointCache/InitialThoughts

The goal of this project is to implement a new Point Cache system based on the [[ https://code.google.com/p/alembic/ | Alembic library ]] and unify the various caching methods used in Blender. This is a prerequisite for further work on particle systems. Having a working Alembic exporter/importer will also greatly benefit integration of Blender into larger pipelines. === Current Issues === * Point cache is used for most physics simulations, but does not include mesh caching. For this there is a separate [[ http://wiki.blender.org/index.php/Doc:2.6/Manual/Modifiers/Modify/Mesh_Cache | Mesh Cache modifier ]], using mdd or pc2 file formats, with its own limiations. Using a unified point cache system would help scene management. * Alembic, unlike the current point cache binary format, supports arbitrary data attributes. This is very important for flexible simulations and user-defined data in particles and mesh data (vertex groups, [[ http://wiki.blender.org/index.php/Dev:Source/Architecture/Custom_Element_Data | custom data layers ]], etc.). * Point cache is limited to fixed-size data sets. This is sufficient for many simulations which only deform geometry (cloth, hair), but for particles and constructive modifier caching variable point data sets are important. * Compression in the point cache file format leaves a lot to be desired. Alembic is specifically designed to have a small memory/disk space footprint and make good use of threading for fast export/import. * Point cache binary format is unique to Blender. While it is fairly simple to interpret, it does not facilitate import/export and integration into larger pipeline workflows. Alembic on the other hand is a commonly used tool and can leverage a large developer community. * Settings for caches are object-centric, each simulation writes its own separate cache files. Baking all caches is simply a loop over all objects, which is not very efficient. Having scene-level settings and operators would make Alembic export/import easier. === Proposal === # Keep basic cache settings on Scene level: - "Disk Cache" vs. "Memory Cache": This could be reframed as having cache file output by default and use ''packing'' in .blend files like we do with images. - Output directory setting for cache files (with nice defaults) - Frame ranges - Automatic cache recording settings (see below) # Optional overrides on object/sim level If deviating settings are needed for individual simulation caches. By default it should not be necessary to tweak each cache individually. # **Question**: How do we handle automatic caching? Current approach is to automatically record point cache when stepping through frames, unless the cache is explicitly baked. This breaks easily when jumping over frames and or scrubbing in the timeline, because simulations need a contiguous ordered frame range for correct results. Does this work well enough to justify keeping it? Are there alternatives to this behavior? One suggestion was to make simulations independent from the animation playback by default, but this makes it difficult to sync simulation results to keyframe-animated objects and to record valid caches on-the-fly. # **Question**: Do we really need multiple caches for each object? This feature adds a list of point caches for every object, only the active cache is used, the others are basically a version control system. Could this be moved to the scene level? === Implementation === {F29943} # Scene stores global settings for caching # Simulations and modifiers can have own settings if needed # Cache data is separate from other DNA by default, but can be packed into blend files like images # Point Cache is a thin API, using Alembic as a backend # Different schema implementations (writer/reader) are created for the various DNA types and to handle interpolation (comparable to current [[ http://developer.blender.org/diffusion/B/browse/master/source/blender/blenkernel/BKE_pointcache.h;885354daec2fa8181aeb27b637c80e4ee4c247df$122 | PTCacheID ]]) # Scene writer/reader ties everything together for exporting the scene as a whole, possibly using object hierarchy (flattening scene structure for export would also be possible) === Notes === * Build instructions for Alembic and the development branch: http://wiki.blender.org/index.php/User:Phonybone/PointCache/BuildNotes * Unsorted list of thoughts on Alembic implementation details: http://wiki.blender.org/index.php/User:Phonybone/PointCache/InitialThoughts * Original [[ https://wiki.gnome.org/Apps/Dia | dia ]] file for the diagram: {F29958}

The goal of this project is to implement a new Point Cache system based on the [[ https://code.google.com/p/alembic/ | Alembic library ]] and unify the various caching methods used in Blender. This is a prerequisite for further work on particle systems. Having a working Alembic exporter/importer will also greatly benefit integration of Blender into larger pipelines. === Current Issues === * Point cache is used for most physics simulations, but does not include mesh caching. For this there is a separate [[ http://wiki.blender.org/index.php/Doc:2.6/Manual/Modifiers/Modify/Mesh_Cache | Mesh Cache modifier ]], using mdd or pc2 file formats, with its own limiations. Using a unified point cache system would help scene management. * Alembic, unlike the current point cache binary format, supports arbitrary data attributes. This is very important for flexible simulations and user-defined data in particles and mesh data (vertex groups, [[ http://wiki.blender.org/index.php/Dev:Source/Architecture/Custom_Element_Data | custom data layers ]], etc.). * Point cache is limited to fixed-size data sets. This is sufficient for many simulations which only deform geometry (cloth, hair), but for particles and constructive modifier caching variable point data sets are important. * Compression in the point cache file format leaves a lot to be desired. Alembic is specifically designed to have a small memory/disk space footprint and make good use of threading for fast export/import. * Point cache binary format is unique to Blender. While it is fairly simple to interpret, it does not facilitate import/export and integration into larger pipeline workflows. Alembic on the other hand is a commonly used tool and can leverage a large developer community. * Settings for caches are object-centric, each simulation writes its own separate cache files. Baking all caches is simply a loop over all objects, which is not very efficient. Having scene-level settings and operators would make Alembic export/import easier. === Proposal === # Keep basic cache settings on Scene level: - "Disk Cache" vs. "Memory Cache": This could be reframed as having cache file output by default and use ''packing'' in .blend files like we do with images. - Output directory setting for cache files (with nice defaults) - Frame ranges - Automatic cache recording settings (see below) # Optional overrides on object/sim level If deviating settings are needed for individual simulation caches. By default it should not be necessary to tweak each cache individually. # **Question**: How do we handle automatic caching? Current approach is to automatically record point cache when stepping through frames, unless the cache is explicitly baked. This breaks easily when jumping over frames and or scrubbing in the timeline, because simulations need a contiguous ordered frame range for correct results. Does this work well enough to justify keeping it? Are there alternatives to this behavior? One suggestion was to make simulations independent from the animation playback by default, but this makes it difficult to sync simulation results to keyframe-animated objects and to record valid caches on-the-fly. # **Question**: Do we really need multiple caches for each object? This feature adds a list of point caches for every object, only the active cache is used, the others are basically a version control system. Could this be moved to the scene level? === Implementation === {F29943} # Scene stores global settings for caching # Simulations and modifiers can have own settings if needed # Cache data is separate from other DNA by default, but can be packed into blend files like images # Point Cache is a thin API, using Alembic as a backend # Different schema implementations (writer/reader) are created for the various DNA types and to handle interpolation (comparable to current [[ http://developer.blender.org/diffusion/B/browse/master/source/blender/blenkernel/BKE_pointcache.h;885354daec2fa8181aeb27b637c80e4ee4c247df$122 | PTCacheID ]]) # Scene writer/reader ties everything together for exporting the scene as a whole, possibly using object hierarchy (flattening scene structure for export would also be possible) === Notes === * Build instructions for Alembic and the development branch: http://wiki.blender.org/index.php/User:Phonybone/PointCache/BuildNotes * Unsorted list of thoughts on Alembic implementation details: http://wiki.blender.org/index.php/User:Phonybone/PointCache/InitialThoughts * Original [[ https://wiki.gnome.org/Apps/Dia | dia ]] file for the diagram: {F29958}