Record and Reporting Delays and Timeouts
Incident Report for Nexmo
Postmortem

What happened

At 2019-06-02 18:52 UTC API requests for call transaction records began to queue and API requests to the Reports API, Verify Search API, Developer API and searching and downloading from Dashboard failed or returned no results. All data was available at 2019-06-02 22:30 UTC. This issue only impacted searching records and did not affect traffic being sent or received.

Causes

Network issues with Nexmo's data storage provider caused by a configuration issue impacted data processing, and traffic was queued until the issue was resolved. The network issue began at 2019-06-02 18:52 UTC and was resolved at 23:17 UTC.

Preventive Actions

A program of preventive work has been identified, including:

  • Improved monitoring and alerting
  • Improved processing of data
Posted about 2 months ago. Jun 26, 2019 - 07:49 UTC

Resolved
The cause of our data outage/delay has been resolved.

The following services were impacted from 02-Jun-2019 18:52 UTC until 02-Jun-2019 22:19 UTC:
Reporting using the Reports API, Developer API and Dashboard
Verify Check API

This data between 02-Jun-2019 18:52 UTC and 02-Jun-2019 22:19 UTC will continue to be unavailable until the data restore is complete, but as of 02-Jun-2019 22:20 UTC all future data is present.

Messaging traffic, Voice traffic and Billing has not been impacted.

All services have been restored. A post-mortem will be published in the next few days.
Posted 3 months ago. Jun 04, 2019 - 09:36 UTC
Update
We are now reviewing data that has been impacted and processing any missing records.
Some records created while the issue was occurring may not be available during this time. We will issue an update in due course.
Posted 3 months ago. Jun 03, 2019 - 09:41 UTC
Monitoring
Our teams have taken corrective actions and the issue will be resolved after monitoring for stability.
Posted 3 months ago. Jun 03, 2019 - 00:06 UTC
Update
We are continuing to work on a fix for this issue.
Posted 3 months ago. Jun 02, 2019 - 20:48 UTC
Identified
Customers are currently experiencing problems impacting delayed reporting and creation of call records.

You may see timeout errors or missing data when trying to obtain reporting information using the Developer API and Reporting API, or performing Verify Search API requests. This began impacting services at 02-Jun-2019 18:52 UTC.

API requests and traffic have not been impacted.

The issue is currently being mitigated. We will update this status as soon as we have more information on this issue.
Posted 3 months ago. Jun 02, 2019 - 20:48 UTC
This incident affected: Verify API and Verify SDK, Developer API, and Reports API.