It factually does not cause loss; it changes the global gain field only, and changes that uniformly for all frames, which is a completely linear and reversible effect. It cannot cause any distortion or degradation; if upon using it, your music actually has audible degradation, one of the following applies:

• The degradation was already present, but you could not hear it because the file was too loud.
• The degradation is caused by clipping, etc., from playing back sound that loud with the system.

In either of these cases, using any other soft-volume control to get the same volume level will reveal the same distortion.

A more likely explanation is the placebo effect. ABX testing is the best way to eliminate that.

Edit: Clarifying my original comment: Nobody wants to do lossy stuff on their music library, regardless of genre.