The code related to 'device_memory' does not handle this situation properly.

Can you elaborate a bit more about it (on a higher level) ?

In D6126#141137, @Sergey Sharybin (sergey) wrote:

The code related to 'device_memory' does not handle this situation properly.

Can you elaborate a bit more about it (on a higher level) ?

For example, let's inspect line 896 of repository device_cuda.cpp:
mem.host_pointer != mem.shared_pointer

Even if mem.host_pointer == mem.shared_pointer, it may have nothing to do with mem state of the current (sub)device. The current device in a multidevice may still have the resource in GPU memory, while some other subdevices in the same multidevice became short of GPU memory, put the resource into AGP memory, and copied the pointer to host_pointer.

This isn't really a higher level overview of the problem.

Am i right that the issue is that multiple compute devices were using same device pointer, while the code was relying on the fact that it never happens?

In D6126#141224, @Sergey Sharybin (sergey) wrote:

This isn't really a higher level overview of the problem.

Am i right that the issue is that multiple compute devices were using same device pointer, while the code was relying on the fact that it never happens?

Sorry, I am trying to elaborate more. The issue is that multiple compute devices in a multidevice share the same mem (device_memory) data structure, (and host_pointer and shared_pointer in it.) The device_pointer of mem is replaced with the actual device pointer for each compute device whenever the multidevice passes mem to it.

Some devices still have enough GPU memory and keep a resource while others put it to AGP memory (and share it). The mem (device_memory) data structure alone cannot reflect the situation like this.

In D6126#141224, @Sergey Sharybin (sergey) wrote:

This isn't really a higher level overview of the problem.

Am i right that the issue is that multiple compute devices were using same device pointer, while the code was relying on the fact that it never happens?

In the current system, mem.shared_pointer is utilized for sharing AGP memory only in the case of multidevice, so the code is written with mem data structure being shared in mind. The point is that it is not possible to determine if a device uses GPU memory only by comparing mem.host_pointer and mem.shared_pointer, ( so cuda_mem_map[&mem].use_mapped_host should be checked first. ) cuda_mem_map[&mem].map_host_pointer in the repository code may be used for that check but 1. the AGP address is already stored in shared_pointer so it is a duplication, and 2. it can be dangled by another device that originally allocated it. This is why I replaced it with a boolean flag and introduced shared_counter in device_memory.

Thanks for the explanation!

Stefan is the original author of the feature, so adding him to the reviewers.

Don't really have time to dig into the code and verify it line-by-line, but overall looks fine. Tested simple scenes and they still work.
More complex scene fails for me, but that also happens in master as well (this is an issue to be solved, but is not related to this patch).