I would suggest having a play with your swapiness settings instead of making your swap spaces smaller.