Inbound Audio Faults & Duplicate Calls
Incident Report for Gradwell Communications Ltd
Postmortem

Description of outage and impact:

Following on from the incident on Saturday, 18th May 2024, we received further reports of inbound calls with no audio. However, this time the issue was slightly different as they were receiving two call legs instead of one. When the call was answered, the call connected and had inbound audio, however, the call continued ringing, and when it was answered again, there was no audio. We saw this impacted the same carrier that had the issues on 18/05/2024 and re-escalated the incident to them directly.

Root Cause & Resolution:

Whilst the fix implemented on Saturday was believed to have resolved the incident, it appeared our carrier missed some media IPs, to resolve this, the carrier migrated the remainder of the media IPs to the new route, ensuring the RTP traffic was routed correctly.

Prevention of recurrence:

We are working closely with the carrier to ensure they adhere to our standards regarding our Change Management process and confirm they notify Gradwell before they roll out any changes to their platform that could impact the way we present calls to our customers. Please accept Gradwell’s apologies for the service disruption and the impact it has had on your business.

Posted May 30, 2024 - 10:01 BST

Resolved
This incident has been resolved.
Posted May 20, 2024 - 13:50 BST
Monitoring
Hello,

Our carrier has resolved the network issue affecting the routing of these numbers. We have conducted a series of test calls, and the numbers are now routing with audio successfully.

If you are still experiencing any issues, please contact our support team on support@gradwell.com. We sincerely apologise for the inconvenience this has caused. We will provide a Reason for Outage (RFO) within 5 working days.

Kind Regards,
The Gradwell Team
Posted May 20, 2024 - 12:38 BST
Identified
Hello,

Our carrier is in the process of rolling out a fix for this issue. Please accept our sincerest apologies for the inconvenience caused.

We will provide a further update at 1:15pm
Posted May 20, 2024 - 12:25 BST
Investigating
Hello,

Following the incident on Saturday, we have identified that the fix implemented by our carrier has caused additional side effects. As a result, customers are experiencing duplicate inbound calls and no audio on some of their numbers. We have re-escalated this issue to our carrier, who have identified the cause and are working towards a resolution.

Please accept our sincerest apologies for the inconvenience caused.

We will provide a further update at 12:45pm
Posted May 20, 2024 - 12:01 BST
This incident affected: Voice & Calls Services (Inbound SIP trunking).