Is your machine perhaps limited in RAM, or running other applications? Handbrake is one of the fancier, more powerful converters out there, and as such makes a lot of demands on a computer.

In my experience, if I avoid Handbrake's more agressive compression settings, I can re-encode a half-hour anime to fit my N900 in roughly 15 minutes. This is on a Core 2 Duo with 4 gigs of RAM.