I've noticed that it feels better when I listen to music that is louder (I'm still trying to figure out why). At the same time, if the music is too loud, then my ears temporarily attenuate to the sound level, and in doing so, they lose some of the clarity they had. They lose clarity because the loudnesses and tones of the notes begin to experience a ceiling effect so that they look more and more the same. That is why, if you have been in a quiet room for a while and start listening to fairly loud music, then it will sound really good for the short time while your ears attenuate to the volume of the music.
So it seems best to strike the right balance between the two effects. However, this position is irritating since you constantly have the feeling that it would sound better if you turned the music louder, but if you do increase the volume, you realize that the lack of clarity begins to ruin the music. So to truly experience the pleasure of the music at the right volume you have to be consciously aback from the those two processes.
Another solution, one already employed in music, is to have songs with a combination of quiet and loud sections so that the person listening will be temporarily unattentuated to the loud music sections. This method of cycling sections with opposite qualities is used for numerous other perceptual and emotional habituations which occur when a person is experiencing any medium.
Another possibility might be to exaggerate the loudness and tone of the song to match a person's attentuation.