Inbound Call Errors affecting Ported Numbers
Incident Report for Gradwell Communications Ltd
Postmortem

On the morning of the 11th of January 2018, a subset of Gradwell customers experienced total loss of inbound call routing. We appreciate that this disruption caused significant frustration to customers, and apologise sincerely for this.

This loss was due to a server failure, causing DNS queries to gradwell.com and gradwell.net not to complete.

We pride ourselves on having redundancies built in to our infrastructure to reduce the risk of service disruption. In the case of this incident, the routing of a subset of phone numbers relied on the resolution of our DNS. On Thursday the 11th of January, the servers performing this action suffered a significant failure that caused all queries to subdomains or hosts on gradwell.com and gradwell.net domains to fail to resolve.

Our agents reacted swiftly to identify the cause of the failure and return service, culminating in restarting the affected servers. While this work was being undertaken the servers were unable to resolve DNS queries, resulting in 28 minutes of service disruption.

Moving forward we have implemented a number of changes to ensure service reliability. DNS for gradwell.com and gradwell.net domains has been moved off Gradwell infrastructure to add another layer of protection to the services and reduce the risk of recurrence of this issue.

Gradwell are more committed than ever to continually improving the quality or our services, and would like to apologise for the inconvenience caused by this incident.

Posted Jan 19, 2018 - 10:25 GMT

Resolved
Our monitoring has confirmed that this incident has been fully resolved, as such this status is being closed.

We will be continuing internal investigations into the cause of the incident, any findings will be post on this status upon completion.

Again we apologise for any inconvenience this may have caused.
Posted Jan 11, 2018 - 12:46 GMT
Monitoring
We have now received confirmation that calls are routing as expected.

If you are still experiencing issues with your calls please contact our support team by email to support@gradwell.com with details of the call behaviour.

Please accept our apologies for any inconvenience this may have caused.
Posted Jan 11, 2018 - 11:52 GMT
Identified
Our upstream supplier has confirmed that they have identified the cause of the incident, and are continuing to investigate internally.

A further update will be post as soon as information is available. Please accept our apologies for any inconvenience this may cause.
Posted Jan 11, 2018 - 11:40 GMT
Investigating
We are currently investigating reports of inbound call disruption to numbers ported from BT. Customers with ported numbers may be experiencing inbound call failures or disruption.

If you are currently experiencing call disruption, please contact our support team by email to support@gradwell.com with the following information:
1. Your customer ID number
2. The numbers(s) affected
3. Any examples of calls that have not connected, including the originating caller number and time of the call.

This incident has been reported to our upstream supplier, and we are currently working with them to resolve. We apologise for any inconvenience this may cause.
Posted Jan 11, 2018 - 11:26 GMT
This incident affected: Voice & Calls Services (Multi User VoIP, Inbound SIP trunking, Inbound IAX Trunking, Single User VoIP).