Page MenuHome
Create Task
Maniphest T61855

Loading times of files that link in external data has become inacceptable
Closed, ResolvedPublic
Actions

Description

System Information
Operating system: Windows 10, latest SP
Graphics card: Quadro K4000

Blender Version
Broken: 358e07f447e9ed7f87f419e8d027e73c97c00077
Worked: f3e9dff03d9bde0d3002eecb1d51dd0baabbb285

Short description of error
The mentioned commit tries to reduce the memory usage when loading linked data. While this may work fine, loading times of such files increased for us in the studio dramatically. As an example, loading time of a file that only links from a single library increased from 2 seconds to roughly two minutes. I am loading both the file and the linked data from a network drive which is write protected. Other files have not suffered that much, but also open way slower than before.

Exact steps for others to reproduce the error
Can't provide a blend as these projects are confidential. The only thing I can post so far is a screenshot of the outliner in 'blend file' mode:

Revisions and Commits

rB Blender
rB779860d34e26 readfile: disable delayed data reading for WIN32

Related Objects

Event Timeline

Rainer Trummer (aliasguru) assigned this task to Campbell Barton (campbellbarton).Feb 22 2019, 4:39 PM
Rainer Trummer (aliasguru) created this task.
Brecht Van Lommel (brecht) added a subscriber: Brecht Van Lommel (brecht).EditedFeb 22 2019, 7:46 PM

I tested .blend load time on a Spring production file on my Linux desktop. Here the change actually improved load time significantly.

BeforeAfter
Local SSD08.28s5.04s
NFS Network19.18s14.04s

This problem may be Windows specific, I've run into extremely slow random access on network drivers there before. Using std::ifstream and seeking instead of mmap helped in that case, apparently due to memory mapping not always going through the system file cache. This might be the same kind of issue.

@Campbell Barton (campbellbarton), maybe we can disable this change on Windows for now, and then investigate how to fix it later? The Linux file load time is actually a great improvement I didn't expect from this change.

Campbell Barton (campbellbarton) changed the task status from Unknown Status to Resolved.Feb 22 2019, 11:37 PM

I'll disable this for windows - for now, this could be tuned by only applying to blocks over a certain size - however if seeking is very slow, this may still be slower.

In my own tests on a HDD (not an SSD) I saw speedups similar to Brecht's, so I didn't think it necessary to tweak.

Campbell Barton (campbellbarton) added a commit: rB779860d34e26: readfile: disable delayed data reading for WIN32.Feb 22 2019, 11:38 PM
Campbell Barton (campbellbarton) added a comment.Feb 24 2019, 10:57 PM

Looks like this is a duplicate for T61880, re-enabling read on demand for WIN32.

Campbell Barton (campbellbarton) mentioned this in rB893fa598319e: readfile: re-enable read on demand for WIN32.Feb 24 2019, 11:05 PM