if the built-in player (gstreamer) fails then try mplayer from devel repository. if you want to have a gui install SiB as well.

there is a discussion on videos on the n900 too -> http://talk.maemo.org/showthread.php...256#post473256

but i have to say that your settings seem really low. in this discussion the videos that have been tried had much higher settings and played nearly smooth.