TTS API degradation
Incident Report for Nexmo
Postmortem

What happened

At 11:57 UTC on 1st March 2019, automated tests indicated an issue with the TTS API for calls using text synthesis in our Washington DC Data Center. Customers with traffic routed through this data centre would have experienced silent calls. After all traffic was switched to the Singapore Data Center at 12:47 UTC, the issue was resolved for customers. A fix for the issue was applied at 13:11 in the Washington Data Centre and all traffic was returned.

Causes

An issue with the underlying servers processing the TTS API requests were unable to synthesise the text.

Preventive Actions

A program of work has been identified that includes improving the Text to Speech service.

Posted about 2 months ago. Apr 02, 2019 - 13:05 UTC

Resolved
The TTS API degradation problem has been resolved.

The TTS API and Verify API calls were impacted from 01-Mar-2019 11:57 UTC until 13:11 UTC.

Voice API and SIP have not been impacted.

All services have been restored.
Please let support@nexmo.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.
Posted 3 months ago. Mar 01, 2019 - 14:52 UTC
Monitoring
We have implemented a resolution for the TTS API Degradation.

We will continue to monitor the service during the next few hours and post updates should anything change.
Posted 3 months ago. Mar 01, 2019 - 12:54 UTC
Investigating
Customers are currently experiencing problems impacting some TTS API calls failing, and also text to speech calls for Verify API. These services have been impacted from 01-Mar-2019 11:57 UTC.

You may see "null" errors when in the callbacks returned from Nexmo, and calls terminating after being answered.

Voice API and SIP have not been impacted.

We will update this status as soon as we have more information on this issue.
Posted 3 months ago. Mar 01, 2019 - 12:40 UTC
This incident affected: Voice APIs (See https://goo.gl/2WoAtx for Current known problems) (TTS API) and Verify API and Verify SDK.