You might try some handle bar mitts ( or muffs)...I have them on the three scooters I have, and they're great! You can get them in waterproof fabric. They usually slip on and off the handle bars easily. I can wear a pair of regular or lightly insulated gloves with them and stay perfectly comfortable.
Karlu
Las cruces, nm