swapping SIM without applying proper procedure is as critical as removing uSD without unmounting and powering down the adapter. You later or sooner will damage your SIM by doing it, the latest when you remove it during a page-erase of the flash storage.
Of course exactly like for any flash memory card (and any storage in general, no matter if HDD, floppy, CD, any cards...) the system has buffers that are not immediately flushed when dirty. Those buffers a) keep info that is not available anymore when card gets removed (see "still can see dir structure of uSD but not open it") and worse b) the dir metainfo may not be consistent, so you at very least see filesystem corruption from removing either uSD or SIM without prior sync. And removing power while flash is written can destroy the storage even physically, definitely will destroy it on low level management metainfo which also often is not recoverable by any means.
Sure swapping of SIM usually works, since the SIM usually is inactive and powered down, even when not sync'ed. But figure you swap the SIM while it's powered down and then the system powers it up again and syncs data in dirty buffers from 'old' SIM onto 'new' SIM. DOOM :-/
There's a reason why systems that allow SIM hotswap have a mandatory SIM-card-detect switch that tells modem about "pending" SIM removal. It's NOT a measure to prevent fraud, that's a silly and naive assumption. It's there to avoid damage done to cards.

bottom line:
DO NOT HOTSWAP SIM OR uSD CARDS! IT WILL DESTROY YOUR CARD.

/j