This looks very promising. Testing on a few models, seems like typically artifacts are gone at subsurf level 1. When there is a big discrepancy between normals and geometry, results are nonsensical with and without this.

In some cases it introduces some darkening, like at the bottom of the monkey's ear with subsurf level 0. I guess this is a case where the position gets pushed into a nearby surface?

I had a hard time finding cases where the Emission Ray Offset Cutoff setting does anything. Do you have a good example of where tweaking is needed? I think we need a way to turn this off entirely, at a minimum for debugging purposes. Not sure if that would be a boolean or could get handled with the same cutoff, since I couldn't get a sense of what the cutoff does.

Some other notes:

• This also affects volume rendering, as points may not be detected to be correctly inside/outside volumes. It may not matter a lot in practice.
• This is only for shadow rays, so artifacts may still be there for indirect light rays. Did you consider adding it for that case as well?
• Ambient occlusion rays may also benefit from this.

You don't actually need to offset the ray on a triangle facing the light (to save calculations). On the other hand, if the cutoff is set to zero, in some cases you can notice on a triangle, that switches from a properly lit to a terminated state, a shadow that suddenly disappears or jumps. To make this behavior smooth and less noticeable, cutoff is used to set fading out width. At the same time there are some ill cases that require offset on a close to be terminated triangles, and that is where the cutoff comes handy.

An example that demonstrates the effect of Emission ray offset cutoff:

Here is an explanation of what the cutoff is. Transmission rays are handled as if the object was inside out (convex becomes concave and vice-versa).

As you can imagine, if you shift the ray way above the surface it can fly over another object and its shadow will disappear. For sure you don't want a tiny infantry on a hilly low poly terrain to have her shadow disappear from under her feet, or you probably want to see a shadow of a X-wing flying close to a low poly Death star. For these cases you can lower the cutoff.

Thanks for the details. I'll run some tests in production scenes to see what the impact is.

As you can imagine, if you shift the ray way above the surface it can fly over another object and its shadow will disappear. For sure you don't want a tiny infantry on a hilly low poly terrain to have her shadow disappear from under her feet, or you probably want to see a shadow of a X-wing flying close to a low poly Death star. For these cases you can lower the cutoff.

Does lowering the cutoff fully solve that, since the offset is still enabled for some cases then?

As you can imagine, if you shift the ray way above the surface it can fly over another object and its shadow will disappear.

@Brecht Van Lommel (brecht) Cutoff=0 completely solves this, while still fixing some shadow terminator artefacts. At the same time cutoff=0 can introduce shadow teleportation on the terminated triangle (which would be in complete shadow if not for the patch). Cutoff>0 on the other hand makes shadows slightly shorter (video).

On the video the 2nd ball has flickering terminator because it is a quadsphere - has non-flat quads and vertex normals don't actually reflect the correct shape. The 3rd ball doesn't have this, because there's still some offset left on those faces. On the 4th ball shadows are significantly shorter. Of course, it's an extreme demonstration - the denser the geometry, the less significant the effect is. My estimation - each subdivision level quadruples number of triangles and reduces ray offset ~4 times as well.

Actually, you were right, the task is much more profound then just adding an offset - ideally one has to consider all the adjacent faces to the incident triangle, the approximation order should vary to work correctly in arbitrary cases, e.g. monkey saddle, concave surfaces have to be lifted above the triangle (to not to induce new ray intersections), but then you have to find a local minima, which is a huge pain even for low approximation orders. Luckily, increasing geometry makes it go away eventually. I haven't tried, but my guess is that smth like D6250: Fix T43835, T54284: Cycles with no ray offsetting can also allow for some improvement here.

I tested on our benchmarks and some production scenes now, and could not find issues. Those generally will not contain low poly meshes with such artifacts, but at least it doesn't seem to break things.

I'm a bit torn on this, because this patch does help, but not reliably in my tests. It eliminates artifacts but also still leaves some in unpredictable ways. In some cases it introduces artifacts that are worse, where you can thin triangle shadows unconnected to the shadow terminator. On the monkey mesh for example I still need to go to subdivision level 3 to eliminate all artifacts.

I feel like this is not the best solution, and that e.g. intelligently ignoring backfaces may work better. On the other hand, this is a working implementation and it does improve the situation. Not sure yet if I want to commit this.

@Brecht Van Lommel (brecht) Thanks for the review! I agree, the patch is anything but perfect - it is hacky, adds additional vague controls and still leaves artifacts.

In some cases it introduces artifacts that are worse

Even if sometimes they look spiky and catchy, in all cases these are smaller and thus can be covered with less shading offset.

intelligently ignoring backfaces may work better

Yes, your approach can improve on this issue, but I don't see how it would solve it completely either - it wouldn't work for non-watertight meshes, for cases where camera is inside and light is outside (like in some weird architectural cases), and probably would still fail for saddle points. The problem with saddles is that shadow ray can intersect geometry more then twice even if the starting point should not be shadowed.

Real models are full of these saddle points:

In my practice I had to increase subdiv by several levels only to fight the shadow terminator bug, but then it requires more memory to render. This patch relaxes this situation a bit.
Anyway, I understand your concerns. So is there any way to battle test this patch in the wild then? Or do we have to wait for alternative implementations to choose from?

The thing with ignoring backfaces is that it's not perfect, but it's more predictable. As a user you can check if the object is a closed mesh or if there are lights inside and fix that, rather than guessing the appropriate subdivision level or factor. Multiple intersections could be handled by counting the number of front/backface intersections, but that does add some cost/complexity to the ray intersection.

Anyway, we don't have to solve that as part of this patch.