the n900 will play many more video formats, with mplayer.
The n8 has better output quality via hdmi, and supports higher resolutions ( up to 720p ) and probably plays all formats you actually need.
Certain videos just cant be played by any of those 2 as the cpu would be to slow and hw accel does not exist for all h.264 profiles, for example.
For (near) hd quality playback on a larger screen, the only choice has to be the n8.
For exotic video formats at relatively low resolution, the choice has to be n900.