We have seen no further issues with our WhatsApp instances. Inbound/outbound traffic has been passed without errors since 12:24UTC.
We will be performing an in-depth post mortem to understand exactly what went wrong and what we can do to prevent recurrence in the future. A report of the results will be published here.
Posted May 31, 2019 - 15:53 UTC
A subset of outbound WhatsApp messages were not sent during this incident... once any individual outbound request had been queued for 5 minutes, it was removed from the queue and an "Undeliverable" status callback was sent to the associated webhook (standard behavior). Therefore those impacted messages were not delayed, but, in fact not sent at all.
Inbound WhatsApp messages are queued for 30 days and therefore were delayed, but eventually sent in all cases.
Posted May 31, 2019 - 14:06 UTC
We have now identified the causative issue within the WhatsApp instances and resolved this problem. Messages are currently being successfully sent and received.
Inbound and outbound messages initiated or in progress during the outage were queued and then forwarded once the service was restored.
Timeline (UTC): 11:50 - 11:54 No WhatsApp messages able to be sent nor received 11:54 - 12:10 Partial recovery, a majority of traffic able to be sent and received 12:24 Service fully available
Posted May 31, 2019 - 13:02 UTC
We have detected an issue whereby outbound Whatsapp messages submitted via the Messages API were not being sent from 11:50 to 12:05 UTC today. We are still investigating, but we can see that traffic was queued and then sent when service resumed.
Only WhatsApp messages were impacted and not other channels available via the Messages API..