It's a problem with the LCD chip or something. (It cannot recieve the info fast enough or something like that)

Use either one of these (linux, and an official windows converter):
http://maemo.org/downloads/PC/

or this (windows, alternative):
http://xoomer.alice.it/sepaolo/n800vc/

and convert your films and all will be fine.