Depends really on the wind but yeah you can drop down a fair bit at times, kinda the same as going uphill. Lighter rollers and a stiffer torque spring help, same as they do for hills, but you will still drop a little.
I tend to do around 42-45mph on flats normally but with an on-coming head wind i have seen me drop to just under 40mph. I used to drop a lot more but i went lighter again on my rollers recently and now can usually keep to around the 40mph mark even in wind, it just takes me longer than normal to get it up to speed if its windy.