If you notice excessive audio latency it's likely because the voice packets being routed through the VSP's server. This is not the most direct route from the remote party to your SIP phone.

Ideally your SIP client will send your public IP address and public port numbers for SIP and voice packets. Unfortunately, if your SIP client does not know it's public IP address it sends your LAN IP address in the SIP headers. The VSP, seeing the LAN address in the SIP header, assumes you are behind symmetric NAT and responds by routing voice packets through their server as this is the only way that works when there is a symmetric NAT router on your end.

Note that a working STUN server is necessary if you want your public IP address in the SIP headers. STUN may not work if you did not set up a STUN server correctly, or you are behind a symmetric NAT router.