On September 15th, 2020, we discovered an issue where some outbound SMS Delivery Receipts (DRs) were missing and not forwarded to customers. There was limited customer impact, affecting DR’s that experienced increased latency only. This issue did not impact SMS traffic.
The issue was mitigated on September 25th, 2020 and a permanent fix gradually rolled out from September 28th to October 7th, 2020.
This issue was caused by a problem with an internal service’s cache size limit. This internal service is responsible for processing SMS DRs. If the cache size reached the maximum limit, the SMS DR “lookup” failed, causing the DR to “drop”, therefore, the DR would not be sent to the customer’s SMS Webhook or SMPP server.