I use either handbrake or mencoder for conversions. Lately I've been using handbrake almost exclusively because it just works. Always. Plus it's multi-platform.
Under Ubuntu I use tablet-encode (you should find it using synaptics), it has a preset for the n900 that works very well.