No inbound audio on calls via a Specific Carrier
Incident Report for Gradwell Communications Ltd
Postmortem

Description of outage and impact:

On Saturday, 18th May 2024, we received reports from our customers experiencing no audio on inbound calls. While the calls appeared to connect, there was no audio present. Outbound calls were unaffected. Upon investigation, we identified the issue with a specific carrier and escalated it directly to them. During their initial investigation, they believed the issue was specific to calls made to a particular route, seeming to affect only ported numbers through BT IPEX. However, as the incident progressed, we observed that it also impacted native numbers associated with the carrier. Gradwell promptly provided examples of the issue to the carrier for their review.

Root Cause & Resolution:

The root cause was identified by the carrier implementing a new route for Gradwell traffic. However, the media IP addresses were not fully migrated, resulting in no audio on inbound calls because the RTP traffic wasn't properly routed. To resolve this, the carrier migrated the media IPs to the new route, ensuring the RTP traffic was routed correctly.

Prevention of recurrence:

We are working closely with the carrier to ensure they adhere to our standards regarding our Change Management process and confirm they notify Gradwell before they roll out any changes to their platform that could impact the way we present calls to our customers. Please accept Gradwell’s apologies for the service disruption and the impact it has had on your business.

Posted May 30, 2024 - 10:01 BST

Resolved
This incident has been resolved.
Posted May 19, 2024 - 10:13 BST
Monitoring
Hello,

Our carrier has resolved the network issue affecting the routing of these numbers. We have conducted a series of test calls, and the numbers are now routing with audio successfully.

If you are still experiencing any issues, please let us know. We sincerely apologise for the inconvenience this has caused. We will provide a Reason for Outage (RFO) within 5 working days.

Kind Regards,
The Gradwell Team
Posted May 18, 2024 - 22:57 BST
Update
Hello,

Our upstream carrier is still investigating this issue. We are continuing to push them for a further update to resolve this.

We will provide a further update on Sunday morning.

Please accept our sincerest apologies for the inconvenience caused.
Posted May 18, 2024 - 21:36 BST
Update
Hello,

Our upstream carrier is still investigating this issue. We'll provide a further update as soon as we can.

Gradwell Apologies for the inconvenience caused.
Posted May 18, 2024 - 20:35 BST
Investigating
Hello,

We are aware of an incident currently affecting a subset of numbers from a specific carrier, resulting in failed inbound audio. Following an update from our upstream supplier, they have identified that this issue is only impacting ported numbers through BT. These numbers are categorised by our supplier as Type A numbers that route through BT IPEX.

The issue has been escalated to BT, and they are treating it as a priority 1 incident. We are actively working on this and will provide further updates as soon as we have them.

Gradwell Apologies for the inconvenience, we are treating this as high priority.
Posted May 18, 2024 - 19:29 BST
This incident affected: Voice & Calls Services (Inbound SIP trunking).