The kernel is an integral part of the software that runs your device. If you remove the kernel, the hardware has no way of communicating with the software. Consider it to be a bridge. If you burn that bridge, nothing comes across. That's why there is a "Remove Power Kernel" application that will remove the power kernel and re-instate the default Maemo kernel before applying the changes.

Don't just remove kernel-power from HAM / FAM, you WILL bork your device.