Personally, I use Handbrake to do video encoding for the N9. It works like a charm and it is open source; you can also create your own profiles so you fiddle only once with the settings.

Handbrake - Open Source Video Transcoder

As a side note, to get the best out of your video transcoding it is a good idea to first have a look at the N9's specifications - it will save a lot of time

Device details - Nokia N9