At 22:30 on 06/09/2018 our network supplier began work to move a number of customers services (circuits) from one network routers to another. This work was in reaction to errors seen on the network on the evening of the 4th, which caused outbound calls for some customers to fail.  
The engineers performing the maintenance were unable to follow the pre-agreed maintenance plan due to additional complications. The decision was made to press on to complete the maintenance despite this complication, to reduce the risk of service disruption.

Due to the complications, part of the maintenance work had to be re-tasked between the engineers. This resulted in a subset of the circuits being moved not being correctly configured on the port of the router they were being moved to. The engineers confirmed maintenance success by checking 20% of the circuits, not all, thus the missing circuits were not identified before the maintenance window closed.  In total, those circuits impacted saw a loss of service for 10 hours.

We have been working with our network supplier since the incident to improve their monitoring of these circuits, and increase the quality of checks performed before maintenance windows close. In addition, we have worked with them to improve the process followed during maintenance work, so that all out of hours faults are identified sooner and therefore resolved before the working day begins. 

We understand how important these services are, and sincerely apologise for the disruption. We are continuously working to improve our services and how we work with our customers. This incident is indicative of how complex any changes can be, and that no change should be taken for granted.

Whilst obviously we want the migration to be devoid of any customer impacting issues, we are committed to the change. The transition will improve the resiliency of our network as a whole, including building additional monitoring, better out of hours support, and improving customer visibility of our services. We believe that customer trust is our best currency, and do not underestimate your continued support of Gradwell. Thank you for continuing to choose Gradwell as your supplier.

Should you have any feedback on this or any service related issue, please contact our support team.