Delivery Receipts Issue
Incident Report for Vonage API
Postmortem

What happened

On September 15th, 2020, we discovered an issue where some outbound SMS Delivery Receipts (DRs) were missing and not forwarded to customers. There was limited customer impact, affecting DR’s that experienced increased latency only. This issue did not impact SMS traffic.

The issue was mitigated on September 25th, 2020 and a permanent fix gradually rolled out from September 28th to October 7th, 2020.

Causes

This issue was caused by a problem with an internal service’s cache size limit. This internal service is responsible for processing SMS DRs. If the cache size reached the maximum limit, the SMS DR “lookup” failed, causing the DR to “drop”, therefore, the DR would not be sent to the customer’s SMS Webhook or SMPP server.

Preventive Actions

  • We have deployed a permanent fix.
  • Improved monitoring and alerting.
Posted Oct 15, 2020 - 08:08 UTC

Resolved
This issue has been fully resolved.
Our Engineering team has finished rolling-out the permanent fix for the missing SMS Delivery Receipts.
Posted Oct 08, 2020 - 22:45 UTC
Monitoring
Our Engineering team has mitigated the issue and are working on a full fix. We will update this status once this is fully resolved.
Posted Oct 08, 2020 - 22:10 UTC
Investigating
We have discovered an issue on our platform.
There is limited customer impact, affecting very few older Delivery Receipts (DRs) of SMS through some of our suppliers. This is not affecting the SMS MT delivery, but only affecting the Delivery Receipts (DRs).
Posted Oct 08, 2020 - 21:53 UTC
This incident affected: SMS API (Inbound SMS).