Voice PSTN Termination - Degradation
Incident Report for Vonage API
Postmortem

What happened

At 09:23 UTC the London Data Center experienced issues connecting with PSTN carriers. The issue was quickly identified and resolved at 09:34 UTC. Customers may have noticed calls failing with a 502 response, or being canceled and receiving a 487. Some calls may have been terminated early due to Session Timers expiring.

Causes

Connectivity on a Network Interface Card in our London Data Centre caused calls to be unable to initiate, and calls did not failover to a different server automatically.

Preventive Actions

To avoid a situation like this from happening again in the future we are planning the following preventative actions:

  • Improved network connection redundancy to PSTN carriers
  • Improved failover strategy for network connectivity issues
  • Improved alerting and monitoring
Posted Jul 01, 2020 - 14:05 UTC

Resolved
The Voice PSTN termination degradation problem has been resolved.

The services were impacted from 25-Jun-2020 09:23 UTC until 25-Jun-2020 09:34 UTC.

Messaging APIs, Verify SMS, and non-PSTN calls have not been impacted.

All services have been restored.
Please let support@nexmo.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.
Posted Jun 26, 2020 - 08:09 UTC
Monitoring
We have seen no further issues since 2020/06/25 09:34 UTC and the issue has been resolved.

The list of services that were impacted has been confirmed as PSTN calls for SIP trunking, Voice API, TTS API, and Verify TTS calls.

We will continue to monitor the service during the next few hours and post updates should anything change.
Posted Jun 25, 2020 - 12:02 UTC
Investigating
We are currently investigating an issue impacting PSTN termination for Voice calls from our London Data Center.

You may have seen 500 and 502 errors when trying to create a call to the PSTN. This will have impacted calls for SIP, Voice API, TTS API, and Verify TTS calls. Ongoing calls have not been affected.

The following services were impacted from 25-Jun-2020 09:23 UTC until 25-Jun-2020 09:34 UTC.

Messaging APIs, Verify SMS, and non-PSTN calls have not been impacted.

We will update this status as soon as we have more information on this issue.
Posted Jun 25, 2020 - 10:13 UTC
This incident affected: Voice APIs (See https://goo.gl/2WoAtx for Current known problems) (TTS API, SIP, Voice API, Europe Traffic).