WhatsApp Service Issue
Incident Report for Vonage API
Postmortem

What happened:

On September 1st, 2021, from 01-09-2021 17:15 UTC to 01-09-2021 18:25 UTC, our Customers may have experienced delayed callbacks for requests originated from the US using our Messages API. All services have been restored.

Causes:
This incident was caused due to new deployments with an internal service that processes callbacks. To be specific after deploying the new libraries we realized that all WhatsApp callbacks that were received in another region but had been originally processed in the US Datacenter were delayed being processed. In order to remedy the situation, we rolled back a recently deployed change. We then also scaled up the number of instances to help with the large queue that had built up.

Preventive Actions:
Improve alerting and monitoring

Improve functional testing

Posted Sep 07, 2021 - 14:32 UTC

Resolved
This incident has been resolved, all services are operational.

A postmortem will be provided with further details in the following days.

Please contact support@nexmo.com if you have any questions.
Posted Sep 02, 2021 - 11:07 UTC
Monitoring
We have implemented a fix for the underlying cause of the delayed/lost callbacks for MT and MOs

We will keep a close eye on/continue to monitor the data services during the next few hours and post updates should anything change.
Posted Sep 02, 2021 - 05:49 UTC
Update
Customers may experience delayed/lost callbacks for MT and MOs. We are still investigating further and more details will be provided as they become available and on the incident's post mortem.
Posted Sep 01, 2021 - 23:06 UTC
Identified
This issue has now been identified. Customers may still experience delays for MT callbacks and MOs sent earlier today. We are still investigating further and more details will be provided as they become available and on the incident's post mortem.

These are some of the current confirmed details.

Messages API: customers may experience delayed/lost callbacks for MT and MOs. We are currently processing delayed callbacks sent earlier today.

The impact is limited to API requests processed in our U.S Data Center.
Posted Sep 01, 2021 - 21:01 UTC
Investigating
Upon further investigation we have found the affected time frame to be between 17:15 - 18:25 UTC. Additionally we have observed that some MOs were also impacted.

We are continuing to investigate the issue and will provide further updates.
Posted Sep 01, 2021 - 18:57 UTC
Identified
We are currently experiencing an issue with our WhatsApp service since 18:09 UTC. Messages API callbacks for outbound messages may be delayed.

Customer impact affects API requests processed in our U.S Data Centers

We will update this incident once we have more information.
Posted Sep 01, 2021 - 18:24 UTC
This incident affected: [Beta] Messages API.