I think this is just how denoiser works and we have to deal with it but maybe it is actually a Bug so here we are.

System Information
Operating system: Ubuntu 18.04

Blender Version
Broken: 2.81 daba3e871ff5 and f2cab8267f22

When using Blender denoise node in the compositor it kill the details like this (zoom in)

this is an original image, after denoise node is applied it looks like this
Notice that all carbon fiber details are gone and I have single color surface instead, I tried using outputs from Denoise pass and also enabling only Diffuse color and Normal outputs and it gives same results even if I only plug in noisy image.

Here is the .blend file

(Note that I wouldn't use denoiser because of these parts but because some others that I removed from file in order to shrink down file size) and there is HDRI packed but I don't know if I did it properly so here is link to it if that matters https://hdrihaven.com/hdri/?h=mall_parking_lot