Well, I guess one man's desperation is another man's heroism.

I was able to track down the performance issues I had in Jessie. For some reason xbindkeys, although having no configuration, caused periodic CPU usage spikes that were that short that they rarely even showed up in top. Since I don't ned xbindkeys in the image I just uninstalled it and everything was back to normal.
I didn't check if a non-empty configuration has the same effect.