Europe datacenter connectivity issues
Incident Report for Nexmo
Postmortem

What happened

On May 7th 2018, between 06:00 and 06:24 UTC, Nexmo experienced a datacenter connectivity problem, impacting ~4000 API requests from customers in Europe and America, on the following services:

  • /search/message API was temporarily down
  • /account/get-balance API was temporarily down
  • Reporting API was temporarily down
  • Account balances were temporarily inaccurate
  • Balance auto-reloads were not triggered
  • API requests to api-eu.nexmo.com experienced timeouts
  • Inbound SIP calls to UK virtual numbers failed 
  • New accounts creation was delayed

Causes

A router hardware malfunction in one of our cloud provider's US datacenter caused a wider connectivity problem between Europe and USA.

Corrective Actions

A program of work has been identified that includes: - Implementation of additional balance/quota service redundancy across datacenters - Optimisation of the internal auto-reload process to address similar scenarios - Improved failover configuration on inbound SIP calls - Reviewing reporting services design to make it geo-redundant

Posted 3 months ago. May 24, 2018 - 09:46 UTC

Resolved
A datacenter connectivity problem on 7th-May-2018, between 06:00 and 06:20 UTC, impacted the following services:

- Account balances were temporarily inaccurate
- API requests to api-eu.nexmo.com experienced timeouts
- Inbound SIP calls
- Developer APIs were temporarily down
- New accounts creation was delayed

All services have been restored. A post-mortem will be published in the next few days.
Posted 3 months ago. May 07, 2018 - 07:18 UTC
This incident affected: Voice APIs (See https://goo.gl/2WoAtx for Current known problems) (Voice API, Europe Traffic) and Developer API, Nexmo Dashboard, Billing and Payments.