I think the idea is fine, but not the specific implementation. I would expect either an empirically determined fudge factor, or an exact error derivation based on the implementation of mean_value_half_tan_v2 and dist_squared_to_line_segment_v2.

I don't think there is a need to go through integers and back to float. This is almost doing 2^log2(x), which equals x. There is some rounding to integers happening, but it's not clear to me how exactly that helps. It usually works to do compute the maximum float value, and then use:

eps = multiplier * FLT_EPSILON * max_value;

Or maybe:

eps = multiplier * FLT_EPSILON * max_ff(max_value, 1.0f);

Where multiplier is a value >= 1 that is empirically determined. It depends on the specific math operations involved. Computing the exact error bound is hard and not really worth it.

Now that you point it out, I also agree that I should probably just skip the log and shifting back and forth.
I was so focused on the formula that I didn't see the simple solution.

Are you sure that you want to just do a simple max_ff without flooring or casting to int?
If not then the fudge factor would not be constant for each precision step, which seems a bit weird to me.

So something like this would make more sense to me:
eps = multiplier * FLT_EPSILON * max_ff(floor(max_value), 1.0f);

Do you think that would be ok?

If you plotted the error per max_value, maybe it looks like a step function. But where those discontinuities are would depend on the specific math operations, they would not be at floor(max_value) in general.

So I don't see the purpose of flooring here.

Maybe it is important to figure out why the current tests are checking for small distances of the point to the line segment. That seems to be just a heuristic someone guessed would make the mean_value_half_tan_v2 errors small enough. But is that heuristic correct? The formula for mean_value_half_tan_v2 is num / denom where num is the difference of two values each of which is the square of a distance, and the denom is a cross product, which also involves a lot of squares of distance differences. So it seems likely that the error is going to depend on how fuzzy are we with squares of distances, not distances. Meaning the heuristic test maybe should have been: is the square of the distance to a line too close to the place where values are indistinguishable, rather than the current code with is testing whether the distance to the segment is being tested for too-closeness. If this is true, this might justify the patches use of a linear function of (FLT_EPSILON * max_value) rather than a linear function of the square of that.

@Howard Trickey (howardt), it may be that by coincidence the micro-optimization for distance comparison and math in mean_value_half_tan match up like that.

But I don't know if the formula in this patch is actually better than squaring the epsilon as we did before, or if it's just the first thing that was tested to work.

I've done some more excessive testing and these are my findings:

Both Brechts suggested values and my initial values seems to work just as well. There doesn't seem to be any difference at all in results.
So I guess we go with the 16.0f one?

Further more, the addition of fabsf(v[i][0] - co[0]) seems to indeed help (at least with the surface deform modifier) when at far away distances.

However, I did notice that we have an other issue here with the surface deform modifier.
At 100m, the binding coordinates seems to be ok. So interp_weights_poly_v2 seems to work.

But when we use the vertex weights to get our position back, there will be floating point precision errors as we add many small and large numbers together.
This happens before this patch, as this is an error with adding up the output values in deformVert in MOD_surfacedeform.c.
It is noticeable even at 1-2m from origin, but the further off, the worse it gets (ofcourse).

But fixing that should probably be for an other review?

Also, should I copy this to the v3 function or should I leave it as is?

Or hmm... It seems like I was wrong there is still precision errors happening at 1m from origin with interp_weights_poly_v2 with the surface deform modifier. Just a lot less severe than before.
Let me check a bit more.

Ok, the errors seems to the there regardless of which epsilon value I choose, so I guess that this is just regular computation errors that we can't side step.
They seems to be in line with the expected error margin.
At 1-2m the error is around 1e-7, which is expected.

Example output values at 1m from origin:

ngon point_v2: 1.00639236 -0.00171839
ngon co_proj: 1.00639224 -0.00171839 0.00000000

point_v2 is the 2d coordinate send into inverp_weights_poly_v2 (which in this case is the same as in 3d space (z = 0).
co_proj is the coordinate derived when we multiply and add the calculated vertex weights together again.

@Brecht Van Lommel (brecht) quick question. Should I commit this for both 2.83 and 2.90 or just 2.90?