I like that idea. If you put a diode in the proper direction from the charger to the USB you could power the USB line to enable USB host mode.
Just plug the charger in when you want to use USB host mode and unplug the charger when you want to connect the USB to a PC.

Sounds like it would work great.

Expanding on that with a switch and a second diode for USB charging as well, I don't feel so good about that. Do you have a SPDT switch in mind? Where would you place it?