HTTP Service Degradation
Incident Report for Nexmo
Update
Our data center provider has informed us that they have found the cause of the issue and are working on a fix with the hardware vendor. The problematic hardware is no longer being used and they therefore do not predict there will be further instability.

At the same time, we are aware that they are operating with reduced redundancy, so all teams will be monitoring systems very closely.
Posted about 10 hours ago. Jun 16, 2019 - 10:56 UTC
Update
We have again experienced periods of higher latency and timeouts from 22:52 - 23:03 UTC and therefore will continue to closely monitor this issue. Mitigating actions are being taken to lessen any customer impact as we work with our data center provider.
Posted about 21 hours ago. Jun 16, 2019 - 00:15 UTC
Update
We have again experienced periods of higher latency and timeouts from 21:42 - 21:46 UTC and therefore will continue to closely monitor this issue. Mitigating actions are being taken to lessen any customer impact as we work with our data center provider.
Posted about 22 hours ago. Jun 15, 2019 - 22:32 UTC
Update
Our datacenter provider has confirmed seeing no further issues from their side since 13:23 UTC. They have further narrowed down the source of the underlying issue, but are continuing their investigations. We are currently working with them to implement mitigating measures in order to ensure minimal customer impact.

Please contact us at support@nexmo.com if you are experiencing any service degradations.
Posted 1 day ago. Jun 15, 2019 - 19:05 UTC
Update
We have again experienced increased latency and timeouts from 1:17pm to 1:23pm UTC and therefore will continue to closely monitor this issue. Mitigating actions are being taken to lessen any customer impact as we work with our data center provider.
Posted 1 day ago. Jun 15, 2019 - 14:10 UTC
Update
We have again experienced increased latency and timeouts from 5:00am to 5:20am UTC and therefore continue to closely monitor this issue. Mitigating actions are being taken to lessen any customer impact as we work with our data center provider.
Posted 1 day ago. Jun 15, 2019 - 07:37 UTC
Monitoring
Our datacenter provider has found root cause and our services are now fully restored.

We will continue to monitor the service during the next few hrs and if you experience any issues please contact us support@nexmo.com.
Posted 2 days ago. Jun 14, 2019 - 21:15 UTC
Update
We are working with our datacenter provider to understand the root cause and awaiting for further updates.

We still see signs of intermittent network issues and therefore will continue to monitor the service during the next few hours.

Please contact us at support@nexmo.com if you experience any issues.
Posted 2 days ago. Jun 14, 2019 - 17:20 UTC
Identified
We've seen a further spike of failed requests from 13:32 - 13:35 UTC. These have been mostly 499s and 504s returned from rest.nexmo.com.

We have now re-routed traffic geographically and are no longer seeing these errors. We are working with our hosting provider to further investigate.
Posted 2 days ago. Jun 14, 2019 - 13:45 UTC
Monitoring
The issue has been resolved and we are following up to fully identify the root cause of the issue.

We will continue to monitor the service during the next few hours and post updates should anything change.
Posted 2 days ago. Jun 14, 2019 - 13:15 UTC
Identified
We are currently investigating an issue that affected traffic in our US Data Center.

You may have experienced errors when making API requests and receiving callbacks.

The following services were impacted 14-Jun-2019:
00:00 - 00:40 UTC - 4.011% of all traffic impacted
04:00 - 04:15 UTC - 0.015% of all traffic impacted
07:00 - 07:20 UTC - 0.028% of all traffic impacted

We will update this status as soon as we have more information on this issue.
Posted 2 days ago. Jun 14, 2019 - 12:14 UTC
This incident affects: SMS API (Outbound SMS, Inbound SMS, SMPP), Voice APIs (See https://goo.gl/2WoAtx for Current known problems) (TTS API, SIP, Voice API), and Number Insight API, Verify API and Verify SDK, Developer API, Reports API, [Beta] Messages API, [Beta] Dispatch API.