I managed to encode movies for the N900 with Handbrake in Ubuntu. Basic pitfalls you want to avoid:

- Use mpeg4, h264-support is rather limited as mentioned above

- the internal player does not support anamorphic scaling! The actual picture size can be whatever you like, but the movie must be encoded with quadratic pixles! (Most DVDs use non-quadratic pixles to get better resolution for widescreen movies, since PAL/NTSC were originally designed for 4:3 aspect ratio only). So go to the picture settings and disable anamorphic scaling!

-Bitrates up to 3500kb/s work fine on my device, which is more than enough for near-dvd-quality film. Higher bitrates may also work, haven't tested since I find it overkill

- Do not use multiple audio tracks! (multiple languages in one file)
- Either use stereo (with dolby pro logic if you want) AAC or multichannel ac3 (with extra codec pack). Multichannel AAC will not work!


no guarantees, these are just my settings that produce good outcome...