Have you tried just following the Quick guide to get a working H264 MP4 file - i.e., not using the presets? I would be surprised if it didn't work, as that is what I am using. It is based on "Normal" profile that loads up when Handbrake loads.

The x264 options should be like this from the command line:

ref=4:bframes=0:subq=6:mixed-refs=0:weightb=0:8x8dct=0:cabac=0

Also make sure that the Modulus is set to 16.

Regarding the "as you like it": it shouldn't cause any problems with Handbake or the N900. You can leave it on Automatic and it will pick the first Audio track (usually English) and go for Dolby Prologic II in the mixdown. You can also add in Subtitles are you want. The reason I left them as optional was that the focus was getting a working H264 video, and the default AAC audio is picked automaticallt by Handrbake as best options. MP3 does work fine as well, and I have tried that in combination of H264+MP3 and Xvid+MP3.

Have you installed anything that could have scewed up Media Player? I had a lot of problems with H264 playback saying exactly what you are getting until I used Web Optimized; it won't play the exact same video file for me, unless that is how the MP4 was encoded.

@Zelig: yeah the movies I was doing were 2.35:1. I am trying to get a balance so I can watch on PS3 as well and not go searching in boxes for the DVDs, so I will probably keep the higher resolution. The size of the file doesn't bother me too much