In my Handbrake with the profile what I've got:
Picture Settings -> Uncheck "Optimal for Source", width: 800 (height will adjust to aspect for movie), uncheck "Keep Aspect", Anamorphic Custom, Alignment 16.
Video: H.264, Framerate: same as source, Constant Quality 20 (I don't like this option and usually do a 2 pass @ 1800-2000)
Audio: English (AAC), AAC (faac), 160 bitrate, 48 SR, Stereo, DRC Off
H.264: Reference Frames 4, Mixed References, Manual Option String "cabac=0:bframes=0:subme=6:8x8dct=0:ref=4", Method Hex, Range 16, Subpixel 6, Analysis Some, 8x8 unchecked.

This works in the N900 in my tests, but some video sources don't work with Handbrake well I've noticed.