Encryption is not the problem, any decent stream cipher is usable.

The limiter here is the bandwidth of the audio connection between subscribers. The codecs normally used to encode/decode the audio limit the usable modulation range and frequencies.
It would need to be determined first what is the usble raw bitrate obtained via the modem algorithms. Note that there has to be quite a lot of redundancy as the encrypted payload gets unrecognizable even with single-bit errors.