Bad form to reply to myself, but here is some more information:

Changing the mplayer video out from vo=omapfb to vo=x11 causes the video to play full screen with the proper rotation, but obviously too slowly to do any good. vo=xv works fine with the screen not rotated, but blows up in portrait mode.

Installing the xrandr command-line program let me see this:

portrait:
Nokia-N800-23-14:~# xrandr -q
Screen 0: minimum 800 x 480, current 480 x 800, maximum 800 x 480
default connected 480x800+0+0 left (normal left inverted right) 0mm x 0mm
800x480 0.0*

landscape:
Screen 0: minimum 800 x 480, current 800 x 480, maximum 800 x 480
default connected 800x480+0+0 (normal left inverted right) 0mm x 0mm
800x480 0.0*

...which I'm not sure how to interpret. The "current" information looks correct, but the info for 0.0 always shows 800x400.

I hope someone can help me out here, because this issue is making me nuts.