VoIP Service Disruption

Incident Report for Gradwell Communications Ltd

Postmortem

On Monday, 20th of March, 2017, we experienced two major service disruptions which resulted in the failure of our VoIP services we provide to our customers and partners. All users of our hosted VoIP and SIP services unfortunately were unable to receive inbound and make outbound calls.

Incident one, which lasted a total of 33 minutes, was caused by a total loss of service of the clustered Asterisk servers. Whilst this was resolved swiftly, further issues were identified which unfortunately caused delays in our service being restored.

After 33 minutes, our engineers performed an emergency update which restored all our hosted VoIP and SIP services.

Following this incident, work has begun to improve the way we monitor our servers. This will improve the alerts we receive before an incident. In addition, resources allocated to the servers which experienced service issue have been increased permanently.

This incident has been defined as major incident due to the far-reaching impact of this outage to our customers.

We acknowledge and understand the significant impact this service downtime had on our customers and apologise unreservedly for the inconvenience caused. Rest assured, we are more committed than ever to continually improving the quality or our services.

Posted Mar 24, 2017 - 16:13 GMT

Resolved

This incident is now resolved, and we can see calls completing successfully.

Once we have completed a full investigation a Reason for Outage report will be posted to this status.

If you are still experiencing issues with your service, please reboot any local equipment including VoIP phones and routers. If this does not resolve the issues please contact our support team on 01225 800888 or support@gradwell.com.

Once again we apologise unreservedly for any inconvenience this incident may have caused.

Posted Mar 20, 2017 - 10:42 GMT

Monitoring

Following work completed by our engineers we now believe this incident to be resolved. This status will remain open for monitoring purposes.

Please accept our apologies for any inconvenience this incident may have caused.

Posted Mar 20, 2017 - 10:19 GMT

Identified

Our engineers have identified the potential cause of the incident and are working to resolve it.

A further update will be posted to our status page by 10:15.

We apologies for any inconvenience this incident may cause.

Posted Mar 20, 2017 - 10:01 GMT

Investigating

Our engineers are currently investigating reports of VoIP service disruption. Customers may see inbound and outbound calls failing.

This issue will be affecting all VoIP services, including IAX and SIP trunks.

Please accept our apologies for any inconvenience this incident may cause. This status will be updated by 10:00 with further information.

Posted Mar 20, 2017 - 09:48 GMT

This incident affected: Voice & Calls Services (Multi User VoIP, Outbound SIP Trunking, Outbound IAX Trunking, Inbound SIP trunking, Inbound IAX Trunking, Gradwell Mobile, Single User VoIP).