U.S.A - Delivery Receipt Latency for Messages Sent Using U.S Short Code
Incident Report for Nexmo
Postmortem

What happened

From April 5th ~16:50 UTC through May 1st ~14:30 UTC, 2019, we experienced periods of intermittent delays with delivery receipts for messages sent using US short codes. Our customers might have experienced delays with receiving delivery receipts when requested.

Causes

One of our US SMS Shortcode suppliers recently completed a data center migration. During this migration, the implementation of a legacy platform setting with a new version of the software caused a bug leading to message queuing on their platform, which prevented their system from sending out delivery receipts in a consistent and timely manner towards their customers. They have now identified the root cause, and have implemented a fix, which has resolved this issue.

Preventive Actions

A program of work has been identified that includes:

--Vonage Preventative Actions:
**Periodic stand-up calls with the supplier to ensure both systems are working as expected and previous issues don't reoccur.
**Working with our internal quality teams to improve monitoring of delivery receipts.
**Introducing KPI’s that measure delivery receipt latency, and trigger alerts if those latencies are above the determined threshold.

--Supplier Preventative Actions:
**Our supplier has updated their configuration management system. Any future version changes (none of which are planned right now), would be run through that management system which would ensure appropriate settings are in place.
**Additionally, They have added monitoring to detect delivery receipt queues. If the queues grow over the expected message limit, this will trigger an alert to their customer care team.

Posted 8 days ago. May 17, 2019 - 19:28 UTC

Resolved
This issue is resolved.

Please let support@nexmo.com know if you continue to experience any problems.
Posted 25 days ago. May 01, 2019 - 04:10 UTC
Monitoring
Our supplier has implemented a fix and are monitoring the situation. We will keep you updated on the progress.

Please contact support@nexmo.com if you have any questions.
Posted 25 days ago. May 01, 2019 - 03:48 UTC
Identified
Our supplier has identified the issue and is working to restore normal delivery speeds as soon as possible. We will keep you updated on any progress.

Please contact support@nexmo.com if you have any questions.
Posted 25 days ago. Apr 30, 2019 - 21:25 UTC
Investigating
Please note that we are investigating an issue impacting delivery receipts in the U.S.A for SMS MT/MO sent using U. S. short codes. Our customers might experience delays with receiving delivery receipts when requested.

Please note that SMS MT/MO messages are not affected by this issue.

Our supplier has been made aware of the issue and is working to restore normal delivery speeds as soon as possible. We will keep you updated on any progress.

Please contact support@nexmo.com if you have any questions.
Posted 25 days ago. Apr 30, 2019 - 17:45 UTC
This incident affected: SMS API (North America (US, Canada & Caribbean) traffic).