I don't think the kernel for 770 does support rotation. I know jott found this link a while ago and is adding rotation support into advanced backlight for 770 users:
http://syslog.movial.fi/archives/7-x...nokia-770.html

(I'd take that to be:
1. Run fb_update_mode auto from http://fanoush.wz.cz/maemo/fb_update_mode.tar.gz
2. Install xrandr from http://sse2.net/rotate/pool/main/x/x....2-0_armel.deb and use xrandr -o left etc