The timer is the cause. The number of task switches is bounded by the timer, and the N900 is at 128 Hz. It happened in my computer, the default kernel is compiled with a 100 Hz timer, and while it has cpu enough for real time xvid capture it stuttered. Once I changed the timer to 1000Hz, even with high cpu load it doesn't stutter. Try building a kernel with CONFIG_HZ=256 and CONFIG_OMAP_32K_TIMER_HZ=256, or even 512 (my DT kernels for the N810 are at 256 with no ill effects at all, and the N810 has a much slower cpu)