You can turn off virtual memory because that is what swap is. Additionally, if you store things in <The name of your device> or any of its subdirectories, it takes up space in your ext partition and not FAT. FAT is pretty much useless, actually. (because you can store anything on ext2 and can't on FAT).