This is normal if the battery is old. Even if you plug the charger the device relies on its battery for power and an old battery would fail on big loads (such as the load during the boot process).

Buy a new battery. There are three good options:

1. An original Nokia battery (too expensive...)
2. A Mugen battery
3. A cheap battery from ebay

If you buy a battery from ebay, make sure it looks like the one on this page. Otherwise it will be too weak (about half the capacity of the original Nokia battery).