Lots of variable here, but you might want to try adding:

canreinvite=no

Then Asterisk won't allow the SIP clients to issue additional invites where they will try to talk directly with each other. (Which can create issues NAT/public addresses, rport, etc)