It appears you can delete some of them using the dpkg tool directly as root. I managed to get rid of the gizmo installer that way. I haven't tried this with the other default applications since it might be tricky to get them back without a reflash.