It's probably starting the kernel and getting the root jffs2 filesystem mounted that's taking time. jffs2 is kinda notorious for long mount times.
~# time mount -t jffs2 /dev/mtdblock4 /mnt/other real 0m4.415s user 0m0.000s sys 0m4.047s