Considering the purpose of the heuristic, I actually think the problem is this min_ff here. Without it, the search would find the actual nearest point with the actual distance (or keep this one if there is nothing), and would then fail on the dist_sq check in the if below. This min basically causes the default result to lie about its distance to the check.

The idea was probably something along the lines of avoiding to look through points that are closer than the initial point but further than the distance limit, but since this actually means the initial point is invalid too. I'd rather do something like:

nearest->dist_sq = len_squared_v3v3(co, nearest->co);

if (nearest->dist_sq > max_dist_sq) {
  nearest->dist_sq = max_dist_sq;
  nearest->index = -1;
}