The current multi-device implementation in Cycles would always duplicate memory for every device. This is generally good for optimal performance since each device will have fast access to a local copy of the data. There are however systems that are able to share quick access to memory between devices. One example is having multiple GPUs connected to each other via NVLink bridges. One such GPU has very fast access to the memory on the connected GPU, pretty much as if it were local.
In these cases it is benefitial to distribute memory across devices, since it reduces the overall memory footprint per device and makes it possible to load larger scenes that would otherwise not fit on the GPU. Two RTX 8000 connected via NVLink can store a 96GB scene this way, whereas right now they could only fit 48GB.
This change modifies the multi-device implementation to support memory distribution across devices. To make this independent of the device backend, the concept of P2P islands is introduced: rather than only having a list of all devices, it now builds a list of connected devices as well. In a system with 1 CPU and 4 GPUs, of which 2 each are connected via NVLink, this would create 3 P2P islands for example (the CPU, the first two GPUs and the last two GPUs). Memory then only has to be allocated once per island, rather than once per device.
The "victor" benchmarking scene for example goes from ~8.3GB per device down to ~6GB on two RTX 2080 Ti.
Since this does not only catch NVLink systems, a new option was added to the settings that toggles whether memory should be distributed (the new additional behavior) or copied for each device (the old behavior). There are system configurations that support P2P, but at lower performance (e.g. P2P without NVLink), so always enabling it would not have been benefitial for those. To avoid cluttering the UI, this new option only becomes visible if the user selects at least two devices that support P2P access with each other.
Example: On this system the two RTX 2080 Tis are connected with NVLink bridge and the RTX 8000 stands on its own:
