The scrolling you are suggesting would be pretty awkward. When scrolling, you are pressing and holding. When releasing the pressure on the screen, you'd want the desktop to stop moving, so you can tap it to launch an app for example. It could work for some but you will have to 'concentrate' on how far you are sweeping instead of the 'easy-going' swipe-swipe, or swipe.

Also, some people want their desktop to be highly flexible i.e. other background for each desktop, or even geeky things like moving desktop (already existing for N900)