I'm not sure if this is the best approach to solve the issues without breaking other things, maybe you have a better solution in mind.

There are already quite separate code paths for dynamic paint and smoke, and I imagined the new particles would be another one like that. So this wouldn't be using the _point methods in PTCacheID.

The abstractions in PTCacheID are not great. The quick solution would be to just extend the if/else in a bunch of places as is done with the various pid->type == PTCACHE_TYPE_SMOKE_DOMAIN tests. I think it's probably better to do that than to put things in extra data?

The proper solution would be to improve the PTCacheID abstraction to have more generic methods like write(), read(), free(), is_valid(), is_empty(), get_info_string(), etc. And then all the stuff specific to handling the number of points and fixed attributes gets abstracted away, and only used for the old particles, cloth and softbody?

In my opinion, ideally PTCacheMem should just store a list of
arrays whereby each array has a size, data type and identifier.
Every simulation type then just has to implement functions that
convert between its state and the list-of-arrays representation.
I'm fairly sure that it is possible to update the other simulation
types to this storage type, but not without breaking existing files.
Therefore, I did not implement this yet.

This seems reasonable to me. I think it makes sense to make the storage for new particles this generic system as you suggest, so that it can be used for more than just particles later.

But we still have older physics systems, and perhaps in the future would want to support volumes or other data structures that may have a more complex data structure or can be compressed in some specific way.

So I'd still make it so PTCacheMem can store data differently depending on the physics system, even if most might end up using shared code.

Instead, I implemented it as a "hack" on top of the PTCacheExtra
"hack"... I reused the custom data types from CustomData, so that
we don't need another run-time type system. I did not implement
disk cache yet, because that makes more sense when I actually got
some ui for it. Without it, testing disk cache is a bit annoying.

I think PTCacheMem could be extended directly to store this list of arrays, rather than using extra data. The way I see it is that members of that like totpoint, data_types, data and cur really belong in a data structure specific to the old particles, softbody and cloth.

If we have generic write(), read(), ... functions each physics system can store data however it wants in PTCacheMem.

I'll probably have to extend the interface of CustomData a bit,
so that I can access information about how a custom data type
is stored internally. This is necessary to avoid the use of MEM_dupalloc
and to get endianness handling correct.

That seems reasonable.

Ok, I'll probably add this as a completely new special case now and refactor it later...

I think we do want to support in-memory/in-blend-file caching as well as on-disk caching. Currently particles support both, but smoke and dynamic paint only support on-disk caching. Therefore, I probably cannot directly use the approach used by smoke.

It seems like we need quite a few callbacks in the general case. However, maybe some of them can be combined. Those come to my mind right now:

• Copy the current state into an in-memory cache.
• Cache the current state on-disk (without first making an in-memory copy).
• Store the in-memory cache in the .blend file.
• Convert the in-memory cache into on-disk cache.
• Inversions of the callbacks above.

Questions for a general solution:

• Can we assume that the cache will always consist of one file per frame on disk? Probably not.
• Can we assume that we always cache one simulation state per frame? We might need to cache subframes at some point, or use something completely different as key than a frame (for example some other iteration count, that is not related to time).