Well, I don't think this is due to swapping. I have a CPU and Memory applet installed which shows the current CPU and Memory used as a bar in the status bar, and neither is near the maximum.