Note that using double doesn't seem necessary, using your code without double precision fixes the bug: P2057.
As far as I can see the this smaller patch could be committed as a fix instead.

source/blender/blenlib/intern/math_vector_inline.c
1191 ↗	(On Diff #35869)	`unit_length` isn't used.

This revision now requires changes to proceed.Apr 6 2021, 9:51 AM

Use float precision

Good to see that the result is also fixed with float.
In fact, I tested this new patch a lot and I couldn't notice a change in the rotation position due to some lack of precision.

Harbormaster completed remote builds in B13889: Diff 35899.Apr 6 2021, 3:13 PM

Germano Cavalcante (mano-wii) retitled this revision from Add double utilities to BLI and refactor 'InputAngle' of transform_input.c to Fix T86889: Rotation with top plane view glitches.Apr 6 2021, 3:18 PM

Germano Cavalcante (mano-wii) edited the summary of this revision. (Show Details)

Campbell Barton (campbellbarton) accepted this revision.Apr 6 2021, 3:33 PM

This revision is now accepted and ready to land.Apr 6 2021, 3:33 PM

Closed by commit rB490801ba1d20: Fix T86889: Rotation with top plane view glitches (authored by Germano Cavalcante (mano-wii)). · Explain WhyApr 6 2021, 3:42 PM

This revision was automatically updated to reflect the committed changes.

Germano Cavalcante (mano-wii) added a commit: rB490801ba1d20: Fix T86889: Rotation with top plane view glitches.