Well the first obvious question is. . . This a personal DVD or a purchased one because the law is still out on making backup's of your purchased DVDs so I won't help you there as that requires breaking macrovision, ect, ect yeah won't go there.

As for personal movies well just keep 90% of the settings same as source and just make sure your output file is 800 x 480. Either format on Handbrake will be fine ( MP4 or H.246 ). Just make sure you don't use super high bitrates and keep the audio a little on the low side when possible ( 128 is fine ) unless you have your N900 OC'd then it can handle a little more with high bitrates..

I think I have my settings something like
900 VBR
30FPS
160Kbps Audio AAC

I don't have my computer with me at the comment so I can't confirm (will update once I know for a fact).