SIP degradation in APAC
Incident Report for Vonage API
Postmortem

What happened

On September 8th, at 6:56 UTC hrs, a hardware fault in our Singapore Data Center affected the SIP services within the APAC region. Customers may have experienced causing network disruption or latency.

Causes

A hardware fault developed on our cloud provider's incoming router in the Singapore Data Center. Our Primary load-balancer failed over to the Secondary and rerouted traffic. Once our vendor rectified the fault, our services switched back to the Primary, however, due to network address caching some services were not immediately restored and required manual intervention. The incident fully resolved at 7:20 UTC hrs.

Preventive Actions

We are making some preventative changes in our failover process and working with our cloud provider to modernise the hardware that supports it.

Posted Sep 18, 2020 - 09:02 UTC

Resolved
This problem has been resolved.

Services were impacted from 08-09-2020 06:56 UTC and 08-09-2020 07:20 UTC.

All services have been restored.

Please let support@nexmo.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.
Posted Sep 08, 2020 - 14:09 UTC
Monitoring
The partial network degradation in APAC is now resolved.

The incident partially impacted SIP services in APAC (SIP trunking, Programmable SIP, calls to SIP URIs). Customers may have experienced call timeouts or SIP 408 responses.

This degradation was experienced between 08-09-2020 6:56 UTC and 08-09-2020 7:20 UTC.

We will keep a close eye on/continue to monitor the voice services during the next few hours and post updates should anything change.
Posted Sep 08, 2020 - 08:33 UTC
Update
We are continuing to investigate this issue.
Posted Sep 08, 2020 - 07:41 UTC
Investigating
Our monitoring has alerted us to a potential service issue with our platform. We are alerting customers immediately while our engineering teams investigate.

If there is a customer impact, it is likely to be affecting our SIP services in APAC since 7:00 UTC.

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://help.nexmo.com/hc/en-us/articles/360015693092-Nexmo-Incident-Handling - for an explanation of our approach to publishing incidents.
Posted Sep 08, 2020 - 07:39 UTC
This incident affected: Voice APIs (See https://goo.gl/2WoAtx for Current known problems) (SIP).