Not sure how this will apply to your NextGen UI, but in terms of the current version, why not have UP and DOWN arrow icons on your toolbar next to the hand. Clicking these would function the same as pressing the keyboard up and down arrows (better still, make it user configurable as to whether these icons mimic UP arrow (pixel pan) or the bigger CTRL-UP arrow pan).

At least then, if you need to pan down, you don't have to go out of whatever mode you are in by selecting the Hand, then pan, then re-select your previous mode - just click on the down arrow (or long press icon to continuously pan down).

(Apologies if this has already been discussed - I haven't read the entire thread).