I have been following your link to OpenBTS
and as I read it the phone radio firmware is probably locked
into needing a BTS to camp on.

The open BTS way sounds completely workable though.

As the radio firmware is probably one of Nokia's darkest secrets
it may be impossible to find a way to actually drive the radio
to to call/sms another radio without going through a BTS.
Just guessing - never really studied GSM before this.

I guess probably the easiest thing to do, if it can be done at all,
would be something like the following:

Pure guesswork here - invitation open for a GSM guru
to step in and correct all this:

1. Monitor the radio for beacons from handphones that can be reached
- build a list of handphones known to be in-range + known handphones (friends).
(how do we do this? I don't see any way to pull that off unless
the radio traffic can be monitored somehow - I suspect not,
but then you never know till you dig into the code.
The unit does scan-monitor for BTS units when signal is lost,
so if it could be tricked into listening for handset beacons, then...)
This is complicated because as I read it the phones are
split off into different frequencies and allotted time slots?
Your list then would be:
cell_number frequency timeslot A-Key?

2. There is probably a transmission header that needs to be
spoofed or perhaps simply copied and pasted
if the intent is to send SMS or transcieve with a unit that
thinks it is logged into a particular BTS or tied to a specific carrier.

3.
There is an encryption handshake that goes on between the
BTS and the handset to establish linkage - not sure yet
whether it is same for voice and SMS
This your handset would need to perform with the other handset.

4. I give up here for the moment.