Page MenuHome
Create Task
Maniphest T93620

Cycles X broke OptiX memory pooling via NVLink
Closed, ResolvedPublic
Actions

Description

System Information
Operating system: Windows-10-10.0.19042-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 2080 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 472.47

Blender Version
Broken: version: 3.0.0, branch: master, commit date: 2021-12-02 18:35, hash: rBf1cca3055776
Worked: (newest version of Blender that worked as expected)

Short description of error
Distribute memory across devies (Optix ) is in Blender 3.0 broken

Exact steps for others to reproduce the error

  • Open the attached blend file in Blender 3.0.
  • Ensure that you have 2x RTX 2080TI with NVLink and enable "Distribute memory across devices" in OptiX

  • Click on render
  • Render will crash with this error:
Failed to build OptiX acceleration structure

OPTIX_ERROR_INVALID_VALUE in optixAccelBuild(context, 0, &options, &build_input, 1, temp_mem.device_pointer, sizes.tempSizeInBytes, out_data.device_pointer, sizes.outputSizeInBytes, &out_handle, use_fast_trace_bvh ? &compacted_size_prop : 0, use_fast_trace_bvh ? 1 : 0) (C:\Users\blender\git\blender-v300\blender.git\intern\cycles\device\optix\device_impl.cpp:1066)OPTIX_ERROR_INVALID_VALUE in optixAccelBuild(context, 0, &options, &build_input, 1, temp_mem.device_pointer, sizes.tempSizeInBytes, out_data.device_pointer, sizes.outputSizeInBytes, &out_handle, use_fast_trace_bvh ? &compacted_size_prop : 0, use_fast_trace_bvh ? 1 : 0) (C:\Users\blender\git\blender-v300\blender.git\intern\cycles\device\optix\device_impl.cpp:1066)

Additional informations

  • "Distribute memory across devices" is working in CUDA mode (Blender 3.0)
  • "Distribute memory across devices" in OptiX is working in Blender 2.93 - you can open the attached file in 2.93 and will see it's rendering.
  • The attached file requires 17GB VRAM in OptiX in Blender 2.93. If you have 2x RTX 3090 this bug can be maybe also reproduced by increasing the required VRAM over 24GB, just duplicate some Suzsannes. I was only able to test it on 2x RTX 2080ti because my lack of 3090s ;)

Thank you for your help

memory test.blend3 MBDownload

Revisions and Commits

rC Cycles
D13488rC2885c4c2c980 Reintroduce device-only memory handling that got lost in Cycles X merge
rB Blender
D13488rB3d5dbc1c4490 Cycles: Reintroduce device-only memory handling that got lost in Cycles X merge
D13488rBe14f8c2dd765 Cycles: Reintroduce device-only memory handling that got lost in Cycles X merge

Related Objects

Event Timeline

Rincewind (Rincewind3D) created this task.Dec 3 2021, 11:56 PM
Rincewind (Rincewind3D) added projects: Cycles, Render & Cycles.Dec 3 2021, 11:59 PM
Rincewind (Rincewind3D) updated the task description. (Show Details)
Rincewind (Rincewind3D) added a subscriber: Patrick Mours (pmoursnv).EditedDec 4 2021, 12:38 AM

@Patrick Mours (pmoursnv)
Maybe you can take a look on this issue?

Patrick Mours (pmoursnv) added a revision: D13488: Cycles: Reintroduce device-only memory handling that got lost in Cycles X merge.Dec 6 2021, 3:31 PM
Rincewind (Rincewind3D) added a subscriber: Patrick M. Jordan (patrick).EditedDec 6 2021, 6:47 PM

@Patrick Mours (pmoursnv)

Hey,

thank you for the quick PR for this issue. But does your PR really fix this issue?
The problem is not getting out of memory, the problem is that the shared memory of two cards via NVLink is not working in OptiX. Memory size should be fine, if memory pooling would work like in 2.93.

Rincewind (Rincewind3D) removed a subscriber: Patrick M. Jordan (patrick).Dec 6 2021, 6:52 PM
Patrick Mours (pmoursnv) added a comment.Dec 7 2021, 11:42 AM

The OPTIX_ERROR_INVALID_VALUE you are seeing is happening because of an out of memory (you can verify this is the log, with "--debug-cycles"). Likely this is the case because BVH builds are happening in parallel, which quickly exhausts available memory because of temporary build memory required (and has some additional known quirks when memory pooling is active), rather than serialized which does not suffer from that problem. That was fixed before (and is in 2.93), but the fix got lost in the Cycles X merge (and thus is not in 3.0), hence why 3.0 behaves differently. The rest of the pooled memory implementation has not changed.

Patrick Mours (pmoursnv) added a commit: rBe14f8c2dd765: Cycles: Reintroduce device-only memory handling that got lost in Cycles X merge.Dec 7 2021, 6:50 PM
Jesse Yurkovich (deadpin) changed the task status from Needs Triage to Needs Information from User.Dec 24 2021, 3:26 AM
Jesse Yurkovich (deadpin) added a subscriber: Jesse Yurkovich (deadpin).

@Rincewind (Rincewind3D) Are you able to try out a 3.1 build to double-check that the issue is fixed?

Rincewind (Rincewind3D) added a comment.Dec 24 2021, 11:06 AM

@Jesse Yurkovich (deadpin)

Yes, it working fine in 3.1.

Tested in:

Blender 3.1.0 - Alpha
December 24, 02:24:18 - 35bd6fe993a1
Philipp Oeser (lichtwerk) closed this task as Resolved.Dec 28 2021, 1:05 PM
Philipp Oeser (lichtwerk) claimed this task.
Philipp Oeser (lichtwerk) mentioned this in T93479: 3.0 Potential candidates for corrective releases.
Philipp Oeser (lichtwerk) added a subscriber: Philipp Oeser (lichtwerk).

So since rBe14f8c2dd765: Cycles: Reintroduce device-only memory handling that got lost in Cycles X merge is in 3.1 master, and it is also on the list T93479: 3.0 Potential candidates for corrective releases, will close.

Philipp Oeser (lichtwerk) added a commit: rB3d5dbc1c4490: Cycles: Reintroduce device-only memory handling that got lost in Cycles X merge.Jan 11 2022, 10:38 AM
Brecht Van Lommel (brecht) added a commit: rC2885c4c2c980: Reintroduce device-only memory handling that got lost in Cycles X merge.Jan 27 2022, 11:37 PM
Pratik Borhade (PratikPB2123) mentioned this in T94530: optix error when trying to render.Apr 27 2022, 7:37 AM