The N9 GPU is the SGX530, a decent general-purpose graphics processor, similar to that used in the iPhone 3GS and iPhone 4 (and, of course, the same as used in the N900). As with those phones, it is just capable of handling 720p (using H.264 baseline profile). (The N900 does apparently need to be overclocked to manage it.)

The N8 uses the Broadcom BCM2727 GPU, a graphics processor specifically designed for running 720p video (in H.264 high profile).

So yes, the N8's video hardware is (in this case) more powerful than the N9's video hardware.