Taking this further (leading to a non-minor refactoring), one could even reduce a lot of redundancy by storing the points' X values separately on group or even action level and just reference it from the fcurves. The idea is that you usually have loc/rot/scale keyframes for all axes at the same frame, which means there's redundancy in the X values for all those fcurves referencing the same animatable object. Furthermore, loc/rot/scale is usually keyframed all together, which means that 9 fcurves share the same X coordinates. Even further, the variation of X values of a whole action is probably very low, or even non-existent, which means the time coordinates of all keyframes of an action can probably be stored in a few arrays of floating point values, while there are many more fcurves compared to the number of distinct keyframe time coordinates. For a large number of fcurves per action sharing the same timing information, this would lead to an overall reduction of memory usage of up to 75%, as fcurves would only store their Y values and reference a shared array of X coordinates.

Ah, didn't know about that page, I've changed the description as best as I could, including the current state of the patch as well as where it will lead to once someone says I should continue development.

The patch's complexity roughly reflects the solution proposed in the description, but the general idea is already in the patch.

After finding out that I misunderstood some properties of FCurves, I got a much much simpler solution.