If one is wearing layers, or they right clothing I'd imagine that there's a percentage of heat that is retained/re-circulated around the body instead of pure heat lost.
I can ride at a slower speed and feel more comfortable, compared to higher speeds where the wind/wind chill makes it more of a hardship.
- Wolf