Yup, I've got the same thing here. From reading the maemo.org archives, it appears that video-out at the highest resolutions may have worked when the device first came out, but for some reason was limited in one of the later Maemo updates. My guess would be this is a work-around for some hardware or software issue they couldn't quite address; after all, smashing HD video down into NTSC requires a bit of effort to do well (or at least a lot of cropping). But who knows...

Well, if you really want to see the high-res video in all its high-res glory, you need to get it off the n900 in a different way. If your TV supports USB input, the n900 works fine as an external drive (this is how I usually do it); otherwise, you'd need to download the video to an HDMI player of one sort or another. The output of the n900 TV-out is, of course, old-fashioned composite NTSC, so it'll never do justice to the actual quality of the video on the phone...

You can control the resolution when you record video. The third icon on the bottom row will let you do this. You can choose between "high", "fine", and "low"; "fine" and "low" videos both still display through the composite tv-out.