Don't forget the memory layout of the N900: 256MByte of RAM and ~740MByte as swap.

With my phone started and all daemons running, I'm left with ~10MByte of RAM and an unused swap. As soon as something memory hungry is started, 10MByte is easily gone pushing some other application out to swap. With a CPU@600MHz and not the fastest buses around, fetching from swap is costly and will take lots of CPU.

If you add things like PulseAudio to this, which easily will gobble down 15% CPU while playing music, you get a pretty sad picture with a great device that's crippled by having to try to handle the same load as a desktop system.