Actually this strengthens my views that it is the CPU/DSP to blame. MPEG4-ASP is more CPU intensive to decode when you increase the resolution. Higher bitrates will also increase the CPU processing required but not to the same extent.

When the device plays 400x240 MPEG4 files, wouldn't the CPU/DSP have scaled the video to the LCD's native resolution (800x480) before it gets to the LCD controller? This would mean that the LCD controller wouldn't know the difference between the 400x240 and 800x480 files.

Also I've just tested my Axim X30H and it was able to play a 624x352 file at approximately 126% normal speed using TCPMP. Since the Axim is in theory much faster (although its hard to compare with a different ARM architecture, DSP, OS and player), it seems likely that the N8x0's OMAP2420 is right on the border of being able to play back this file without a hitch.