Voice API degradation
Incident Report for Vonage API
Postmortem

What happened

At 12:35 UTC on March 17th 2020, customers started to experience issues generating Voice API requests, missing voice callbacks and failed calls.

Services were impacted from 17-03-2020 12:35 UTC until 17-03-2020 13:00 UTC and from 17-03-2020 15:20 UTC until 17-03-2020 19:08 UTC. Traffic was rerouted to an alternative data centre to restore service.

Causes

Network latency and packet loss caused by an issue with our Dallas data centre provider's network infrastructure.

Preventive Actions

A program of work has been identified that includes improving the failover of Voice services within the Vonage API platform.

Posted Mar 30, 2020 - 17:04 UTC

Resolved
This incident has been resolved.

Please be advised Voice API traffic has now been switched back to our U.S Data Center.

Services were impacted from 17-03-2020 12:35 UTC until 17-03-2020 13:00 UTC and from 17-03-2020 15:20 UTC until 17-03-2020 19:08 UTC

Please let support@nexmo.com know if you continue to experience problems with this.

A post-mortem will be published in the next few days.
Posted Mar 18, 2020 - 10:19 UTC
Update
Please be advised outbound Voice API requests are still failed over to our European Data Center.

However, customers should no longer experience issues generating Voice API requests, missing callbacks or with calls failing.

We will continue to monitor the service and post updates should anything change.
Posted Mar 17, 2020 - 16:59 UTC
Update
After an extended period of monitoring, some customer Voice API requests and calls may have continued to experience issues. Therefore, please be advised we have failed over outbound Voice API traffic to our European Data Center.

We will continue to monitor the service during the next few hours and post updates should anything change.
Posted Mar 17, 2020 - 16:03 UTC
Monitoring
Customers may have experienced issues generating Voice API requests and modifying in-progress calls using our U.S Data Center.

Customers may have experienced 500, 502 and 504 errors, failed calls and missing callbacks.

Services were impacted from [17-03-2020 12:35] UTC until [17-03-2020 13:00] UTC.

We will continue to monitor the service during the next few hours and post updates should anything change.

Please let support@nexmo.com know if you continue to experience problems with this.
Posted Mar 17, 2020 - 14:58 UTC
This incident affected: Voice APIs (See https://goo.gl/2WoAtx for Current known problems) (Voice API).