Service Issue in Platform - Singapore Data Center
Incident Report for Vonage API
Postmortem

What happened

On July 30th at 13:28:59 UTC, customer API traffic routed to our Singapore Data Center started to experience increased latency or timeouts due to issues affecting our Content Delivery Network (CDN) provider.

Causes

Our CDN provider experienced a localized outage in the Asia-Pacific region.
This caused some customer API requests originating from this region to the api.nexmo.com and rest.nexmo.com endpoints to experience increased latency or timeouts.
Our systems first alerted us of a potential problem at 13:34:00 UTC and we were able to confirm the root cause and mitigate any impact at 14:07:00 UTC.

Preventive Actions

Improved alerting and monitoring.
Review our CDN redundancy approach to minimize the disruption of similar events may have to our customers in the future.

Posted Aug 07, 2020 - 09:40 UTC

Resolved
This problem has been resolved.

Services were impacted from 2020.07.30 13:28 UTC - 2020.07.30 14:07 UTC.

All services have been restored.

Please let support@nexmo.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.
Posted Jul 30, 2020 - 16:00 UTC
Monitoring
We have identified and implemented a fix for this issue.

Services were impacted from 2020.07.30 13:28 UTC - 2020.07.30 14:07 UTC.

Customers in the Asia-Pacific region may have experienced timeouts or increased latency reaching the API endpoints rest.nexmo.com, api.nexmo.com and accessing the Vonage API Dashboard.


We will continue to monitor the service during the next few hours and post updates should anything change.
Posted Jul 30, 2020 - 14:57 UTC
Investigating
Our monitoring has alerted us to a potential service issue with our platform. Although It is not yet certain whether there is any customer impact, we are alerting customers immediately while our engineering teams investigate.

If there is a customer impact, it is likely to be affecting customer traffic in our Singapore Data Center only.


We aim to provide an update on our investigation within 1 hour.

Please see this article - https://help.nexmo.com/hc/en-us/articles/360015693092-Nexmo-Incident-Handling - for an explanation of our approach to publishing incidents.
Posted Jul 30, 2020 - 14:25 UTC
This incident affected: [Beta] Conversation ([Beta] Conversation API), SMS API (Outbound SMS, Inbound SMS, SMPP, Asia & Oceania traffic), Voice APIs (See https://goo.gl/2WoAtx for Current known problems) (TTS API, SIP, Voice API, Asia & Oceania traffic), and Number Insight API, Verify API and Verify SDK, Nexmo Dashboard, [Beta] Reports API, [Beta] Messages API, [Beta] Dispatch API.