I keep it at 4 since I believe this to be the best all purpose rear shock setting when considering the fact that my girlfriend is sometimes on the bike with me, and I have a very large top case which often gets filled with 30-50 LBS worth of items.
You could probably get away with 3 but I think 4 is a better all around shock setting if you do not wish to change it every time you have a heavier load on it.