It seems to be related to file size. The torrent that was killing me was 2gb. I paused it, added a 170mb torrent, still very unresponsive even at extremely 15kb/s.

I removed the 2gb torrent, and the 170mb torrent ran at 150kb/s without any lagging at all.