Hmmm small status update. So far I've done the following:

installed and setup pySerial with no problems (needed to install python2.5-dev)
Found there was no need for alien/rpmbuild (found source for debian python-wxgtk2.6)
Tried to build APSW and got a million syntax errors in gcc :P I'll try in scratchbox instead of on the device.

Oh and I found that the Debian Armel already has a ready-to-go package for bitpim which makes me wonder why I'm trying this crazyness anyway. Could someone who's got either the debian chroot or full debian on their device perhaps install bitpim and try it?