Voice API and In-App Voice calls terminating in London Data Center
Incident Report for Vonage API
Postmortem

What happened

On 11th November 2020, we discovered an issue where some Voice API and In-App Voice calls processed in our London Data Centre experienced media timeouts, causing calls to be terminated unexpectedly. There was limited customer impact between 09:35 and 10:21 UTC, affecting a small number of calls processed in our London Data Centre. This issue did not impact other SIP traffic.

Causes

This issue was caused by an internal network routing error during routine maintenance, causing media traffic not to leave the Vonage API platform. After 30 seconds of no media traffic, the call is terminated by the Vonage API Platform.

Preventive Actions

  • Enhanced monitoring and alerting
  • Improved deployment processes
Posted Nov 20, 2020 - 14:18 UTC

Resolved
The Voice API and In-App Voice calls terminating in the London Data Center problem has been resolved.

The following services were impacted from 2020/10/11 09:35 UTC until 2020/10/11 10:21 UTC.
• Voice API in London Data Centre only
• In-App Voice in London Data Centre only
• Programmable SIP in London Data Centre only

SIP and Messaging APIs have not been impacted.

All services have been restored.
Please let support@nexmo.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.
Posted Nov 12, 2020 - 09:22 UTC
Monitoring
We have implemented a fix for the media timeout for Voice API and In-App Voice calls in our London Data Center that caused calls to terminate.

The list of services that were impacted has been confirmed as:
• Voice API in London Data Centre only
• In-App Voice in London Data Centre only
• Programmable SIP in London Data Centre only

A small number of calls may have terminated unexpectedly after experiencing silence.

We will continue to monitor the service during the next few hours and post updates should anything change.
Posted Nov 11, 2020 - 11:26 UTC
Investigating
Our monitoring has alerted us to a potential service issue with our platform. Although It is not yet certain whether there is any customer impact, we are alerting customers immediately while our engineering teams investigate.

If there is a customer impact, it is likely to be affecting Voice API and In-App Voice calls in our London Data Center.

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://help.nexmo.com/hc/en-us/articles/360015693092-Nexmo-Incident-Handling - for an explanation of our approach to publishing incidents.
Posted Nov 11, 2020 - 10:34 UTC
This incident affected: [Beta] In-App IP communications ([Beta] In-App IP Voice) and Voice APIs (See https://goo.gl/2WoAtx for Current known problems) (Voice API).