The power issue aside, a colleague of mine recently had a major house fire, started by a mobile phone charger. The phone wasn't connected at the time, but the charger was still plugged in.

I know the probability of this is small, but when you can rule it out completely for very little effort, why take the chance? May as well unplug it - if it's accessible, of course.