The way I solved this problem was I unmapped some IMAP folders from Gmail. You can do this by enabling "Advanced IMAP Controls" in Labs.

However the problem is it will not synchronized the unmapped folders, so this may not be what your looking for.

The folders I mapped was (inbox, drafts, sent, trash) and it is very fast for me since it does not try to synchronize the other folders, just the ones I have mapped.