Ok, solved the problem. Maybe somebody else will come to this problem so I quote what Johan Hedberg wrote me (thanks Johan!):

"You can use the maemo libopenobex with Bluetooth but you need to use FdOBEX_TransportSetup and give it a connected RFCOMM socket. The BtOBEX* functions are not available in the maemo version of openobex since including them would mean linking with libbluetooth wich in turn would mean that libopenobex (and all applications linked to it) would become GPL (instead of LGPL which is its intended license)."

This works now for me.