If you are using Windows XP (although I think that Win7 suffers from the same problem as well), it has a nasty habit to cache USB drivers for fast startup when you connect a device to a specific USB port.