Well, like you said... the screen is only 800x480... so the best screen resolution to play on the N900 is clearly 800x480. That said, 800(h) by 480(v) means you best play vids that "fit" into these dimensions. In the N900 case, 480p would be the best. If that is not available, go for the closes lower resolution. In the original poster's case, 360p. 1080p is usually 1920x1080 and to fit that into the N900, the program has to "scale down" (ie. unnecessary work, albeit the slowdowns mentioned) and should just use something close to native screen resolution (800x480).

Anything higher or lower resolution and you will lose detail due to the source or screen limitations. So the best would be 360p. I hope I understand your question and explained it correctly.