Page MenuHome
Create Task
Maniphest T97510

Noticeably higher RAM consumption with packed textures
Closed, ArchivedPublicBUG
Actions

Description

System Information
Operating system: Has been reported on Mac and Linux, has been observed under Windows as well
Graphics card:

Blender Version
3.1.2 but has been reported as early as January 28

Short description of error
For context, I am reporting on behalf of the SheepIt Renderfarm team.

Our users and we noticed a significant increase in RAM consumption and IO when using packed textures as compared to externally linked textures.
In a test a with a 3 GB Blend file 151GB were read from disk.
The report in January was about a project that could be rendered on a 16 GB Mac with linked textures but would use up to 60 GB when packed.

I will include their final verdict:

I’ve found the issue.

I went through intensive testing, I compressed the textures as best as possible and that made the memory peak go down from 49 gigs to about 20 gigs… that’s pretty good but not much of a solution.

I then found out that what causes most of the massive memory spike lies in Blender’s unpacking algorithm.
So I unpacked the file, turned on Relative Path and zipped the folder with all the files. This made the build time BLAZING FAST (we’re talking mere seconds) and the memory peak dropped to… 3 gigs!!!

We cannot provide you with any of these files so I tried to recreate it. In my tests relative.blend peaked at 850 MB while packed.blend used 1380 MB. I understand that in this case the difference could just be the extraction of the the textures into RAM so I am not really satisfied with this example. I will try to provide a better one if need be.

Exact steps for others to reproduce the error

  • Unzip attached file
  • Open relative.blend
  • Check RAM memory consumption
  • In File → External Data click in Pack Resources
  • Check RAM memory consumption
  • (Optionally you can load the packed.blend file to skip a few steps)

or

  • Add objects with PBR materials.
  • Case A: Reference to the textures locaton
  • Case B: Pack them into the blend file using File -> External Data -> Pack Resources
  • Start rendering

bug report.zip472 MBDownload

Event Timeline

Harlekin (randomGuyOnTheInternet) created this task.Apr 21 2022, 1:53 PM
Harlekin (randomGuyOnTheInternet) updated the task description. (Show Details)
Raimund Klink (Raimund58) added a subscriber: Raimund Klink (Raimund58).Apr 21 2022, 2:01 PM
Germano Cavalcante (mano-wii) changed the task status from Needs Triage to Needs Information from Developers.EditedApr 22 2022, 6:08 PM
Germano Cavalcante (mano-wii) added a project: Core.
Germano Cavalcante (mano-wii) added a subscriber: Germano Cavalcante (mano-wii).

I can confirm the extra RAM consumption even though the images are not actually being used in Blender.

Memory is allocated in BKE_packedfile_new.

And right after it is written in the Blender file in BKE_packedfile_blend_write.

If data is written to the .blend file, and the packed image is not being used, should it really consume RAM?

If this is a bug or intentional, I imagine it's up to the Core team to inform us.

Germano Cavalcante (mano-wii) updated the task description. (Show Details)Apr 22 2022, 6:12 PM
Harlekin (randomGuyOnTheInternet) added a comment.Apr 22 2022, 6:21 PM

Thanks for taking a look. I should have specified that I mean RAM consumption during rendering specifically, not during the packing

Harlekin (randomGuyOnTheInternet) updated the task description. (Show Details)Apr 22 2022, 6:22 PM
Bastien Montagne (mont29) moved this task from Backlog to Blender 3.2 on the Core board.Apr 25 2022, 12:21 PM
Bastien Montagne (mont29) changed the subtype of this task from "Report" to "Bug".
Bastien Montagne (mont29) added a subscriber: Bastien Montagne (mont29).

Extra RAM consumption with packed files is expected, by definition you put the external file's content inside your current Blend file. We could maybe theoretically try to 'clear' the embedded file once the Blend has been written on disk, and have a mechanism to reload it when required, but that would likely be complex to get working properly, since it would be violating quiet a few rules/principles in current data-blocks and blend file handling.

But issue here is related to rendering, and I think this is because CoW copies of IDs (from depsgraph evaluation) currently blindly also duplicate the packed files data! This is obviously wrong, will prepare a patch for this.

Bastien Montagne (mont29) added a subscriber: Sergey Sharybin (sergey).Apr 25 2022, 5:23 PM

hrrrmmm, investigating the code further, images are actually NOT duplicated in depsgraph... Nor during rendering it seems.

So besides the first point in my first comment ("Extra RAM consumption with packed files is expected").

Regarding rendering, when using Cycles there is an optimization for external files, which get directly read by Cycles into its own image buffers, completely by-passing Blender's Image ID /ImBuff system. So if you do not load the image in Blender itself (by e,g displaying it in the Image editor or the 3DView). This is not possible currently for packed Images, which need to be first read by Blender into its own cache (ImBuff buffers system), and then stored again by Cycles into its own data structure, resulting in the image being three times in memory (one as packed data, one as Blender cache, one as Cycles image). Thanks @Sergey Sharybin (sergey) for helping understanding what's going on here.

While this is not optimal, there is no bug here. Thanks for the report.

For the record, here are my measurements with provided test file:

Render engineBefore RenderingRenderingDifference
Cycles linked:0.4 G1.6 G1.2 G
Cycles packed:0.9 G2.6 G1.7 G
Eevee linked:0.4 G3.4 G3.0 G
Eevee packed:0.9 G3.9 G3.0 G
Bastien Montagne (mont29) closed this task as Archived.Apr 25 2022, 5:24 PM