You've got a couple of options with the new handbrake.

If you're just targeting stock codecs, you'll need to use mp4 containers (and remember to turn on the web extensions if you're going to stream!). Inside that you can use the basic divix mode, but it's much better to use h.264.

Start with the iphone profile and then make a few tweaks to take advantage of the n900's extra power.

1) You can raise the number of ref frames to 4 safely, and probably 5.

2) You won't need to scale down the video to iphone resolutions, but you will need to resize, because the default media player can't handle non-square pixels (which is what all DVDs have).

Resize 4:3 DVDs to 640x480. For 16:9 DVDs, it's a little trickier. You can expand them to 848x480 (or 1024x576 for PAL) but that's higher than native. I'd suggest 800x448 which is what the media player will use anyway.

3) For bitrate, you have two choices. If you're only going to use the files off a memory card or the internal memory, then use constant quality mode and use something around 20. If you're going to stream the file, you will be limited by the media player's broken buffering. In my tests, I found that a the overall rate (audio + video) must be around 1Mbit/s at the most. After subtracting audio (assuming 128kbps), that leaves 865 for video. So use fixed rate at 865 with two pass to make the most of what you've got.

4) For audio, transcode to aac.

5) If you're using the extra codec pack, then you can use mkv containers and keep ac3 audio tracks, but those will be decoded with the cpu, wasting battery, and you won't get surround sound, so it's pretty pointless.

Hope that helps