Setting "Auto Snap" to "Nearest Frame" currently behaves in such a way that it both limits movement of keyframes to whole frames (not subframes) and merges two keyframes when they're both occupying the same frame.

However when selecting keyframes in the dopesheet and scaling them down it only limits the movement of the keyframes to whole frames but doesn't merge keyframes when they're both occupying the same frame.

When wanting to fix this and make sure it behaves in the same way then of course the question arises: "How does the code know which keyframe is more important, which one should it keep and which one should it throw away?"

My suggestion would be to simply split the difference. Check the values for all the competing keyframes in a single frame and go with an average. Given the "Auto Snap" current behavior in Blender this feels intuitive from the user's point of view. Assuming it will auto snap to the nearest frame only to find out in the graph editor that it created a tangled mess of keyframes occupying the same frames is in no way user friendly behavior.

If the user would not want any snapping they always have the option of turning it off, that's never been a problem.

I'm including a simple Blend file that has two cubes with simple Z-location animation. The keyframes for one of them have been scaled to 66% (with Auto Snap set to "Nearest Frame") and in the Graph Editor you should see how the keyframes have been snapped to whole frames but not merged together. You can scale down the keyframes of the original cube in order to replicate the function.