Ah, thanks for the updated info. When swap-to-file was introduced it was an issue (and at least through the 2.2 series, not sure about 2.4 because I didn't use swap-to-file there).
Do you know if this is still true with a FAT32 file system? In principle it should, because an application like the LILO loader does the same kind of mapping you describe. But in an earlier post skeezer reported that there was a noticable difference ("way faster").