Issue affecting Customer Dashboard and Numbers API
Incident Report for Vonage API
Postmortem

What happened

On February the 25th at 12:58 UTC, our monitoring services alerted us to an issue impacting one of our database servers in Europe. Our engineering team immediately investigated the alerts and re-routed the traffic to a working server at 13:47 UTC. Services started to recover shortly after and the problematic database server was fixed and put back into rotation at 14:29 UTC. This caused customers not being able to log into the customer dashboard, and API calls failing with internal server errors for the following services: MMS sent via the Messages API, Numbers and Account API and Reports API. The impact on Customer Dashboard started at 12:00 and the rest of the services started at 13:00. Impact finished at 13:47 UTC.

Causes

One of our machines became unresponsive which caused a slowdown in our cluster and eventually caused the cluster to become unresponsive

Preventive Actions
Upgrade our machine with a newer software version.
Implement a new database performance monitoring tool
Improve periodic testing alerts and monitoring processes.
Enhance alert escalation processes to assign engineering resources more efficiently

Posted Mar 11, 2021 - 11:38 UTC

Resolved
This incident is resolved. We will publish a post portem in the next few days. If you have any questions, please reach out to support@nexmo.com
Posted Feb 25, 2021 - 16:18 UTC
Monitoring
Our team has found the root cause of the problem and has applied a fix. We are going to monitor these services, and we will update this post if anything changes. Normal service should be restored.
Posted Feb 25, 2021 - 14:08 UTC
Update
This incident also affects the Reports API. Customers would see API calls failing for Reports API endpoints.

We will post and update as soon as more information becomes available.
Posted Feb 25, 2021 - 13:57 UTC
Update
Customers are currently experiencing problems impacting Customer Dashboard, Numbers API, Account API and Messages API (MMS only). These services are impacted since 13:00 UTC.

We will update this status as soon as we have more information on this issue.
Posted Feb 25, 2021 - 13:48 UTC
Investigating
Our monitoring has alerted us to a potential service issue with our platform. Although It is not yet certain whether there is any customer impact, we are alerting customers immediately while our engineering teams investigate.

If there is a customer impact, it is likely to be affecting the following services: Customer Dashboard and Numbers API.

We will post an update as soon as we know more
Posted Feb 25, 2021 - 13:17 UTC
This incident affected: Developer API, Nexmo Dashboard, [Beta] Reports API, and [Beta] Messages API.