Issue with API services
Incident Report for Vonage API
Postmortem

Date/Time Impacted

2022/01/11 11:00 UTC until 2022/01/11 12:27 UTC

Services Impacted

  • Customer Dashboard
  • Developers API
  • Subaccounts API

Summary

From 2022/01/11 11:00 UTC until 2022/01/11 12:27 UTC, we experienced degraded performance across multiple services due to a discrepancy in the database. This resulted in issues with user logins and sign-ups for the customer dashboard and also had a knock-on effect on other services.

Customer impact was mitigated by restarting the database cluster and ultimately all the services were restored after the restart was successfully completed.

Product-specific impact:

Customer Dashboard: From 2022/01/11 11:00 UTC until 2022/01/11 12:27 UTC customers may have experienced loading the customer dashboard.

Some customers were not able to log in and sign up. For logged-in users, certain sections of the customer dashboard would have been usable (Numbers, Applications, etc.) but not profile and payment-related features.

Developers API: From 2022/01/11 11:00 UTC until 2022/01/11 12:27 UTC customers may have experienced slow / failing responses.

Sub accounts API: From 2022/01/11 11:00 UTC until 2022/01/11 12:27 UTC customers may have experienced slow / failing responses.

Root Cause

Due to a database discrepancy, a lock was created in the dashboard schema which prevented further SQL queries from being executed.

Restoration Action

We restarted the database cluster and ultimately all the services were restored after the restart was successfully completed.

Next Steps

  • Improve the alerting process
  • Improve database Schema
  • Implement regional failover
Posted Jan 12, 2022 - 16:50 UTC

Resolved
The issue with the API services and customer dashboard has been resolved.

The following services were impacted from 11-Jan-2022 11:00 am UTC until 11-Jan-2022 12:27 pm UTC:
Customer Dashboard
Developers API
Subaccounts API

No other services have been impacted.

All services have been restored.
Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.
Posted Jan 11, 2022 - 13:41 UTC
Monitoring
We have implemented a fix for the issue impacting APIs and the customer dashboard.

The list of services that were impacted between 11-Jan-2022 11:00 am UTC and 11-Jan-2022 12:27 pm UTC has been confirmed as:
Customer Dashboard
Developers API
Subaccounts API

We will continue to monitor the service during the next few hours and post updates should anything change.
Posted Jan 11, 2022 - 12:43 UTC
Investigating
We are currently investigating an issue impacting our services.

Customers may experience issues loading the customer dashboard. Also, customers may experience issues with developers API and subaccounts API.

The following services are impacted since 11 am UTC:
Customer Dashboard
Developers API
Subaccounts API

No other services have been impacted.

We will update this status as soon as we have more information on this issue.
Posted Jan 11, 2022 - 11:55 UTC
This incident affected: Developer API and Nexmo Dashboard.