Page MenuHome

Better handling of temp files
Closed, ResolvedPublicDESIGN

Description

Right now, Blender just stores its temp files in temp directory. In some cases, it even uses PID to avoid several instances of it using the same files (at least, for point caches of unsaved .blends, and added that behavior to fix T39521, see rB6cd717e0a3c6).

The issue is that Blender never removes those files, and some OSs (Windows…) never clean up their temp dir, which can leads to a quick waste of space. So here is the proposition (also from discussion with Campbell):

  • At startup, Blender creates a (PID-tagged) folder in temp dir, where it stores all files that are really temp ones.
  • When closing, Blender removes that folder and all its content.

For the very few cases where we need a 'cross-session' persistent 'temp' files (only render result afaik?), we could create temp file directly in main tmp dir (not in Blender session’s PID-tagged one).

The issue here is how to avoid several instances to use the same 'persistent' file? We could use locks, but I can’t find a nice way to handle them (main issue being to handle crashes, which are one key reason to have those persistent render results). Or maybe we just decide this is a corner case we do not want to support (i.e. in situations like T39521, we simply accept both processes write the same temp render result file - intermediate samples being saved in PID temp folder anyway)?

Event Timeline

I suggest to look at what other applications do, this must be a common problem. There also exist operating system functions like mktemp to get unique filenames, which I think we should use when possible rather than creating our own system with PIDs.

One case to think of as well is crashes, you don't want all the temp files to stick around when Blender crashes.

For the render result we should make the caching an option I think, that is disabled by default. Not sure about the best location for the file, maybe the output file path. It can still create a temp file if the caching is disabled.

I'm not so much fun of this PID and mktemp things actually.

Not sure what else apart from the saved buffers gives issues, but for saved buffers it'll be better idea to NOT use PID but use full .blend file path converted to a proper file name (i.e. /home/sergey/blah.blend becomes /tmp/home_sergey_blah.blend). This way it'll be still possible to restart blender and use render buffers from the previous instance.

Just an idea, if we wouldn't find it's suitable then i'll fine with the PID salt or mktemp. Maybe mktemp is the better approach in fact.

Not sure what else apart from the saved buffers gives issues, but for saved buffers it'll be better idea to NOT use PID but use full .blend file path converted to a proper file name (i.e. /home/sergey/blah.blend becomes /tmp/home_sergey_blah.blend). This way it'll be still possible to restart blender and use render buffers from the previous instance.

That’s already what is done, but this fails when you render *the same frames of the same file* in two different processes (like in org BT, where the only diff was the used cameras).

For the render result we should make the caching an option I think, that is disabled by default. Not sure about the best location for the file, maybe the output file path. It can still create a temp file if the caching is disabled.

Agree, current situation is a bit like bug-that-becomes-a-feature…

One case to think of as well is crashes, you don't want all the temp files to stick around when Blender crashes.

Yes, this is the main issue, yet have to check whether our crash handler (the one writing crash log) would be able to also delete temp files (hopefully it will ;) ).

Bastien Montagne (mont29) changed the task status from Unknown Status to Resolved.Jun 23 2014, 1:58 PM

Applied D531 as rB414c70435dcd5.