A while ago, i think it was on the IRC channel, there was someone talking about how they had used a SIP server (i think it was running on a N900, though there might have been a laptop and/or a WiFi router involved) to call other SIP+WiFi enabled devices, i think this was done while camping, creating their own little SIP network while they were there.
I have been following your link to OpenBTS and as I read it the phone radio firmware is probably locked into needing a BTS to camp on. The open BTS way sounds completely workable though. As the radio firmware is probably one of Nokia's darkest secrets it may be impossible to find a way to actually drive the radio to to call/sms another radio without going through a BTS. Just guessing - never really studied GSM before this.