Most likely some process is allocating memory all the time without letting go of it. This forces the kernel to put other processes in the swap to make room for the running ones.

In your XTerminal, run the command top and press m to sort by memory usage and check for processes that use a lot of memory. It's also good to look for processes that use a lot of CPU.