If you want smooth playback you have to transcode videos if they aren't in right format for standard player. It plays videos with DSP and more optimized for ARM while mplayer plays all with CPU. I use MPEG4 and MP3 codecs and avi or mkv containers. 800x450@25 fps and 3Mbit/s plays fine. I think you don't need better quality for this screen.
I played a 192Kb/s MP3 song with standard player and CPU load was about 40% @250MHz. Mplayer did the same with 70% @500MHz (through pulseaudio). And note that N900 don't use DSP for audio. All of this is just low level optimization for particular CPU.