On Windows (in write through mode, the default) the only thing safely remove does is ensure you don't have any application reading or writing to the device (which would mean you'd lose data).

Most other operating systems use some sort of write caching, which means that you lose information every time you eject it without unmounting first.