I had to change a few more places than I've thought at first, unfortunately.
This seems to fix the behavior for Workbench, Eevee and Cycles.
The patch can be tested with this file. It uses a temporary change in the Mesh to Volume modifier that makes it output a single voxel every time.
The voxel should be centered at the world origin:



