Go to ~/.modest/cache/mail/pop/ (in case it's a POP3-account) and list the directories there. Each corresponds to one mail account.

(You may find directories for accounts you already deleted in modest. Remove them.)

Within each of these directories there's a folder 'cache', which in turn contains a number of strangly named sub-directories ('05', 3a' etc.). I only deleted these sub-directories as I wasn't sure what I was doing. It might be safe to remove the 'cache'-directory itself, but I don't know for sure.