Sometimes.
After longer rides at high speeds, without ear protection, everything sounds weird and muffled for a while. I figured that's a sign I damaged my hearing, so I started wearing ear plugs.
The exhaust noise isn't loud at all at those speeds, but the wind is.