VoIP Phone Registration Errors
Incident Report for Gradwell Communications Ltd
Postmortem

On Monday, 20th of March, 2017, we experienced two major service disruptions which resulted affected customers using our hosted VoIP service, who register using the NAT proxy. This unfortunately meant that customers couldn’t register phones, or make and receive calls.

Incident two, which lasted a total of 18 minutes, was caused by several errors appearing on our NAT servers. Our NAT servers play a key role in our network, and these errors affected new VoIP phone registrations as well as existing devices that have been registered.

After 18 minutes, our engineers updated the server’s resources. This immediately allowed new VoIP phone registrations, with tests confirming the issue had been resolved.

Further to this incident work has begun to ensure this incident does not reoccur in the future. This includes improving the alerts we receive before an incident like this happens, as well as the resources being increased permanently to the servers.

An investigation has also begun into the servers automated responses and why the server did not respond to the resource problems in the same manner as the SIP registration servers. It’s important for us to understand internally why there appears to be a difference, but at time of writing, the reason remains unclear.

This incident has also been defined as major incident due to the far-reaching impact of this outage to our customers. We acknowledge and understand the significant impact this service downtime had on our customers and apologise unreservedly for the inconvenience caused.

Rest assured, we are more committed than ever to continually improving the quality or our services.

Posted Mar 24, 2017 - 16:15 GMT

Resolved
Following remedial work completed by our engineers we now believe this issue to be resolved.

If you are still experiencing issues with your service, please reboot any local equipment including VoIP phones and routers. If this does not resolve the issues please contact our support team on 01225 800888 or support@gradwell.com.

Once again we apologise unreservedly for any inconvenience this incident may have caused.
Posted Mar 20, 2017 - 12:09 GMT
Identified
Our engineers are currently working to resolve this issue. The incident is not affecting all services as initially believed, it will only be affecting customers who register their devices using our NAT proxy.

A further update will be posted by 12:15. We apologise for any inconvenience this incident may cause.
Posted Mar 20, 2017 - 11:59 GMT
Investigating
We are currently investigating reports of issues making and receiving calls, and errors registering VoIP phones with our service.

Customers will not be able to make or receive calls. An update will be posted on this status by 12:15.

Please accept our apologies for any inconvenience this incident causes.
Posted Mar 20, 2017 - 11:54 GMT
This incident affected: Voice & Calls Services (Multi User VoIP, Outbound SIP Trunking, Outbound IAX Trunking, Inbound SIP trunking, Inbound IAX Trunking, Single User VoIP).