Yeah, I kinda gave up looking for a good preset a while back. There are some floating around, but I generally just mess with the settings myself.

I haven't played with my settings in a while. Let's see... On my current version of Handbrake (0.9.5 on OS X), starting with the "normal" preset, you should already have the video codec set to "H.264 (x264)" on the "video" tab. The first thing to do is to go to the "Advanced" tab, and set things up for a baseline profile: this means setting "Maximum B-Frames" to 0, switching off "CABAC Entropy Coding", and turning off "Weighted P-Frames". This should be enough to start with. I know that videos encoded using this setup will work fine in media player on my N900.

After that, things are really up to you. Use the "Picture Settings" icon at the top of the window to set the video dimensions to whatever you want (presumably to something within the range of your phone's native resolution!). You can manage the amount of compression in a variety of ways on the "video" tab, and the "advanced" tab has even more ways to tweak the compression (at the cost of letting Handbrake take longer to do its job). Both the MP4 and MKV file formats should work fine, choose whichever you prefer.

There are some resources availabe here at Maemo.org. In particular, the wiki has its own video encoding page (covering Handbrake and a number of other encoding options):

http://wiki.maemo.org/Video_encoding

There are a number of Handbrake-related threads floating around as well, though I don't think any one of them is a perfect resource. Anyway, a search for "Handbrake" here should turn up some good stuff.