Verify API - Outbound SMS
Incident Report for Nexmo
Postmortem

What happened

Between 12:32 and 15:26 UTC December 5th, all Verify transaction flows that included SMS were interrupted. The outbound SMS steps of the Verify flow were not initiated due to an internal failure. The subsequent text-to-speech voice calls for any Verify request were successfully generated however.

Causes

At 12:32 UTC on the 5th of December, a new deployment was made for the Verify API product. This new patch modified the length of an internal HTTP request which then caused all related internal SMS API calls to fail.

Preventive Actions

A program of work has been identified that includes:

  • Exposing additional metrics and setting up alerts based on the metrics to help identify such issues earlier.
  • Adding additional smoke tests and relevant items in deployment checklists.
  • Introducing additional post deployment checks within application logs, specifically specific HTTP errors.
Posted 2 months ago. Dec 12, 2018 - 13:21 UTC

Resolved
This incident has been resolved.
Posted 3 months ago. Dec 05, 2018 - 22:29 UTC
Monitoring
Between 12:32 and 15:26 UTC, a number of Verify transaction flows were not fully completed. The outbound SMS steps of the Verify flow were not sent out due to an internal failure. The subsequent calls for the Verify request were successfully generated, however. This resulted in a lower rate of success for Verify requests during this period.

We believe this is now fixed, however, we will continue to monitor the performance of the Verify product to ensure that the issue has been fully resolved.
Posted 3 months ago. Dec 05, 2018 - 16:11 UTC
This incident affected: Verify API and Verify SDK.