You are talking about unmounting, not physically removing the microSD?
Removing the back cover will not unmount the microSD automatically.

I have my swap file on microSD, no impact on removing the back cover itself.

And losing swap is not the worst thing that can happen to your system.