Test file from T97259:

	Embree	CUDA (BVH2)	OptiX
Old
New

As a general note regarding self intersection, it feels like that should be a solved problem in the literature. But for example in the relatively recent A Fast and Robust Method for Avoiding Self-Intersection, the proposed solution is to use the same ray offset mechanism that we used before.

It relies on object and camera transforms being applied to mesh vertices to be robust. In final renders we already apply object transforms, and we could consider doing the same for camera transforms though that would add a fair amount of complexity to shading. However none of that works for instancing as in the test file, and would have poor performance in interactive 3D viewports.

There is also the case of exactly overlapping faces that we don't handle very well right now, though the same artifacts will appear in the 3D viewport with rasterization, so in that sense it's relatively predictable. We could potentially solve that by rejecting intersections at the exact same distance as the current triangle (perhaps by increasing tmin to exactly 1 bit beyond it). But this again relies on being able to match the triangle intersection bit-for-bit.

CC @Patrick Mours (pmoursnv) in case you have any interesting ideas for OptiX here.