System Information
Operating system: Windows-10-10.0.19041-SP0 64 Bits
Graphics card: GeForce RTX 3090/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 456.71

Blender Version
Broken: version: 2.91.0, branch: master, commit date: 2020-11-25 08:34, hash: rB0f45cab862b8
Worked: n/a

Short description of error
I am creating an ocean in Houdini, and export displacement map as EXR sequence for rendering in Cycles. The EXR images contain RGB channels for vector displacement, and an Alpha channel for Cusp(white water). I used the color channel for displacement and it has no problem. However, when I use the Alpha channel to adjust the color, the displacement suddenly becomes huge, in both viewport and final render. Scaling it down will not fix the problem because the interpolation of the displacement is also altered. I have tried with both 16bit and 32bit EXRs and the problem remains. I had this problem in previous Blender versions, and different PCs(different CPU/GPU) too.

Exact steps for others to reproduce the error

1. Create a plane, add Subdivision Surface modifier, check Adaptive Subdivision.
2. Create a new material, add Displacement node, add Image Texture node.
3. For Image Texture node, open an EXR image file with RGBA channels.
4. Plug color output of Image Texture to Displacement node, and Displacement to Material Output.
5. Leave Principled node plugged into Material Output.
6. Add a Color Mix node, and set two different colors(e.g. black/white), plug color output to Principled color input.
7. Plug the Alpha output of Image Texture into Fac of the Color Mix node.
8. Add a Colorramp node. // This is where my problem shows, the displacement in the render view suddenly changed(becomes huge on Z axis).
9. If problem does not occur, continue to use Image Texture Alpha or Color output as factor to play with the color.

// This problem happens at a probability(about 90% in my case).
My Temporary Fix

1. Disconnect Principled node from Material Output.
2. Disconnect Displacement node from Material Output.
3. Disconnect Principled node color INPUT from any node.
4. Replug Principled node into Material Output.
5. Replug Displacement node into Material Output.
6. Replug Principled node color INPUT to previous color nodes.

// If I tried to connect/disconnect or add other nodes at this point, the same problem will come back again.