Hmmm, is it?

I might turn up the usleep delay in the loop. Come to think of it, the printf I stuck in there to report how much it was delaying was probably delaying itself. I'll count how many cycles that loop actually goes through (I know usleep(5) isn't right...).

Is the CPU usage higher when you're not touching anything?

Edit: the system seems to sleep a little longer than 5 microseconds. That loop is not executing more than once or twice, and even then only immediately after when you put the mouse down to the next tick. Are you sure it's this?