Description of outage and impact:

On 1st September 2023, Gradwell Engineers were alerted to some customers experiencing outbound call failures. Gradwell Engineers were fully engaged in the incident which was given a priority one status.

Cause & Resolution:

Our engineers identified the cause of the failures and deployed the changes across our platform at 7:59am.

Root Cause:

The root cause of the outbound call failures was due to a permission constraint on our Call Detail Record (CDR) database table, most significantly the CDR table for September 2023. This impacted outbound calls for customers who had a minute bundle on their account. Customers who did not have a minute bundle on the account at the time of the incident were not impacted at all.

Prevention of recurrence:

Gradwell will be expanding our regular outbound call test to include calls that are included in bundles, ensuring early detection in case of recurrence. Additionally, we will be enhancing our alerting system to promptly detect any issues with generating our CDRs for upcoming months. Please accept Gradwell’s apologies for the loss of services and the impact it has had to your business

Posted Sep 06, 2023 - 15:23 BST

Resolved

Hello,

Following Gradwell's monitoring of the incident, we are confident that the services have been restored successfully.

Please accept our sincerest apology for the inconvenience caused. We will provide an RFO within 5 working days.

Posted Sep 01, 2023 - 08:30 BST

Monitoring

Hello,

We have implemented a fix and we can see the call traffic returning to normal. Please retest and call our support teams if your calls are still experiencing any issues.

Gradwell apologises for any inconvenience and disruption this has caused you and wants to thank you for your patience.

Posted Sep 01, 2023 - 08:05 BST

Identified

Hello,

We have identified the cause of the failures and we're in the process of implementing a fix.

Please accept our sincerest apologies for the inconvenience caused. We will provide a further update by 8:15am.

Posted Sep 01, 2023 - 07:56 BST

Investigating

Hello,

We are aware of outbound call failures this morning. Please be advised this has been escalated to the highest priority & our teams are working to restore the service as soon as possible.

Gradwell apologies any inconvenience and disruption this has caused you and want to thank you for your patience whilst we investigate and resolve this incident.

We will provide a further update within the next hour.

Posted Sep 01, 2023 - 07:24 BST

This incident affected: Voice & Calls Services (Multi User VoIP, Outbound SIP Trunking, Single User VoIP, Wave).