What's the resolution and bitrate of the video? What's the video encoded in?

From what it's doing it looks like either the bitrate or resolution might be too high for the n900 to process.

You can try going to the Extras repository and download the package "Extra Decoders Support" although I dont think this will solve your problem.

You could download an alternative video player to see if it works although it wont provide acceleration. You can download mplayer (cli movie player) and sib (mplayer gui frontend) and see if it'll play the video file.