AVI is not an indication of what codec the video is using, it's only a container. The shuttering means that the video you are trying to play has too large bit rate or it is encoded in a format that is too CPU intensive for N900 to handle.

Only things you can do are:
1. Overclock, most units can go up to stable 800Mhz
2. Play with Gstreamer based player eg. stock media player. It uses the DSP to accelerate playback
3. Re-encode video