This happens on larger selections for vertices 'inside' the selection
which have all of their edges set to a -1 index (at the start of
'bmo_inset_region_exec'). This would result in a division by zero (in
'bm_edge_info_average_length').
Relevant previous commits:
rBe387ccdb4270
rB41a5e731a2e1
rBf48efbf4642e
Not particularly sure which exact case rBf48efbf4642e was trying to
solve, but this is where it started.
Previous methods of calculating average edge length for inside verts as
well was also not working well, so what I tried here is reach out from
the inside vert until a valid vert is found on the boundary [excellent
excuse to have a look at bmesh walkers :)] and use its average edge
length instead. This results in a pretty solid looking inset, not sure
if this is really the purpose of "Offset Relative" option.
If this is all bogus, the least we should do is return 0 or 1 and not
return NANs...