On September 1st, 2021, from 01-09-2021 17:15 UTC to 01-09-2021 18:25 UTC, our Customers may have experienced delayed callbacks for requests originated from the US using our Messages API. All services have been restored.
This incident was caused due to new deployments with an internal service that processes callbacks. To be specific after deploying the new libraries we realized that all WhatsApp callbacks that were received in another region but had been originally processed in the US Datacenter were delayed being processed. In order to remedy the situation, we rolled back a recently deployed change. We then also scaled up the number of instances to help with the large queue that had built up.
Improve alerting and monitoring
Improve functional testing