Now I have an N900 I've messed with handbrake for AVC/264 and found that the N900 doesn't like the following:
B frames (the default doesn't work - set to zero)
8x8 DCT (not selected by default)
CABAC Entropy Encoding (selected by default - so deselect)

Otherwise it seems to eat up the handbrake settings.