Declare the function static, so that the compiler knows it shouldn't be an "exported symbol" -- that way any other C file can also declare a function with the same name, and they won't conflict with each other.

Remove "This function is used to"; generally such comments are written as an imperative ("Loop over the keyframes...").

The fact that the function loops is not interesting (at this level), though, as it's pretty much an implementation detail. I think the comment can be removed, and the one at the declaration in the header file can be enhanced a little.

This function only ever returns 0, so just make it void and don't return anything.

There is only one call to this function, and that passes key_ok=NULL. This means the parameter can be removed, as it's not used anyway. This'll make the code a lot simpler. Yes, it will be less generic, but it's never a good idea to add code that isn't used.

Configure your editor to remove trailing whitespace on save, and to use autoformatting with Clang Format. Or run make format before submitting a patch, but that'll take longer and will be more of a hassle (and easier to forget).

This will get eKeyframeVertOk values; since it's a bitwise combination of those enum values and not an enum value itself, ok cannot be declared as of eKeyframeVertOk type, but at least add a comment that references eKeyframeVertOk.

This approach leads to lots of code duplication. Since this is run in an operator, and not on every frame change or something like that, it's fine if it's not running at 100% optimized speed. Simple, easy to understand code has less bugs, and is easier to adjust later when requirements change.

Move this code into the conditional that deals with the left/right sides only when they're enabled. That way it only computes the angle when it's actually going to be used.

Never compare enum values to their underlying value directly, but use the symbolic names (EQUALIZE_HANDLES_LEFT etc.)

Combined with the note I left at the enum declaration (to ensure the left/right values map to individual bits), you can change this condition to:

if (mode & EQUALIZE_HANDLES_LEFT) { ... }

Looking at this code, amount seems to be the desired handle length. This means that the code doesn't "equalize" per se (which I would expect to investigate the current handles, and make sure that they are equal somehow), but rather just sets the handle length. handle_length would thus be a better parameter name than amount.

I'd use bitfields for this, so that one bit is "left" and the other is "right", and when both are set it's naturally "both".

EQUALIZE_HANDLES_LEFT = (1 << 0),
EQUALIZE_HANDLES_RIGHT = (1 << 1),
EQUALIZE_HANDLES_BOTH = (EQUALIZE_HANDLES_LEFT | EQUALIZE_HANDLES_RIGHT),

Document the parameters. Some are quite clear, so not all of them need documenting, but at least amount and flatten should be described.

Declare mode as type eEditKeyframes_Equalize

Rename ..._types to ..._sides; these do not indicate "types of equalization" or "types of handles", but simply the left/right sides.

Remove the trailing period (also from the other two descriptions); Blender adds them automatically.

The description shouldn't just copy the name. "Ensure all handles have equal length, optionally making them all horizontal" would be a better one IMO, as that describes what's being done without having to know what "equalize" or "flatten" mean in practice.

Why the default of 5.0?

A zero value won't work well; handle points shouldn't be placed directly on the keyframe point itself. A minimum length of 0.1f will be fine. For the soft minimum, maybe even 1.0f would be nice? I'd certainly set the soft maximum a bit lower too, as it determines the range of the slider, and too high a maximum will make it overly sensitive and thus harder to use.

I did a quick analysis of a shot of Sprite Fright, and these were the results:

avg_length: 0.747
max_length: 9.0
min_length: 0.175
90% percentile: 1.33
95% percentile: 2.0

This means that 90% of the handles had a length of 1.33 or less, and 95% had 2.0 or less. I think having a soft max of 2.0 would likely be fine.

Add descriptions to these parameters to explain what they do.

The only returned value is false, so just declare as void and don't return anything.

The ked parameter was only there to pass to the key_ok function, which has been removed, so ked and all the code manipulating it can also go.

Declare at the use, and not here at the top of the function.

ok isn't used.

Flip the condition and use continue:

if ((bezt->f2 & SELECT) == 0) {
  continue;
}

This allows you to unindent the rest of the code.

This is tricky, as the tense in that sentence suggests it's the current handle length, rather than the resulting handle length. "Desired handle length, must be positive." looks better to me.

This should use the LISTBASE_FOREACH macro.

Move the declarations here, instead of declaring at the top.