So that's exactly where the problem starts.
Do you think you could fix that? Don't bother about powering VBUS for now, it's not that simple, not that important, and probably best managed independently of the logical switch to hostmode (imagine hostmode and nevertheless powering device via USB, by using an external Y-cable)

cheers
jOERG