Sorry I should have read your post a little better - this may or may not work, it depends on what the DSP is on the N8 (whether it uses different instructions, different DSP kernel and additional coprocessors, etc). But probably not is the answer.

I also note, after looking at the Nokia N8 page, that it only claims 720p decoding anyway.

Edit: As above, for some reason I thought we were talking about 1080p - so yes, the N900 hw should be able to decode 720p with well written drivers and a fast enough clock speed. But I think you'll need to provide the drivers...