We will post here again upon completion of this emergency work. We apologise in advance for any problems caused by this.

***Update 13:54***

We have fully powered off the problem blade and are restarting services now. Some virtual machines will require a filesystem check so these will take slightly longer to complete.

***Update 14:12***

We have now restored all services. If you are still seeing any problems, please do raise an incident with our support team and we will investigate on a case by case basis. Our apologies again for the loss of service.

No user intervention will be needed as all current SQL databases will be migrated by our system admin team during the above window. Only users with databases being served from mysql5db-1.gradwell.net will be offline, all other database users will be unaffected.

If you are unsure of where your SQL databases are, please log in to your hosting control panel and click on ‘Manage Databases’ This will list all of your current databases and the server on which they are located. We apologise in advance for any problems caused by this downtime.

***Update 21:20***

Due to the amount of time taking to dump the necessary databases, and double check the dumps contain all the data, it would appear we will need more time than initially expected. Therefore we are unable to complete the works tonight. We will reschedule this maintenance for Friday night and will post here tomorrow with an updated window.

This will be affecting some of our customers websites and home directories while we attempt to restore full service again.

***Update 14:24***

This issue was cleared in full yesterday shortly after posting the initial status update. We have been closely monitoring the storage since then and have seen no further errors.

We are now happy to close this status notification and we apologise for any problems this might have caused you.

Whilst the whole network will be ‘at risk’ for the entire period, the main windows for major network outages are specific per device. Our work plan will be as follows:-

Core Switching - 01:00-2:00

ADSL Routers - 02:00-02:30

Border Routers - 02:30-03:30

Peering Routers - 03:30-04:00

We apologise in advance for any loss of services caused by this and if you experience any problems after this work has been completed, please contact support@gradwell.com

*** UPDATE 07:05 ***

We are investigating a problem with the web cluster following this maintenance.  Status will be updated here as soon as further information is available.  We apologise for any inconvenience caused.

***Update 13:38***

We have been monitoring all upgrades and they are performing as expected. All work was successfully completed. We apologise again for any loss of service during these upgrades.

Update 05:51 : This problem was first detected by our monitoring systems just after 4AM.  We have confirmed that this is likely to cause VoIP registration issues for some customers.  Our network engineers are working with our transit providers to restore connectivity.  We are not able to provide an ETA for restoration of our link with Sovereign House at the moment.  We apologise for any inconvenience this will be causing customers are are are working to ensure the link is restored by UK business hours.

Update: 06:14: We have opened an internal issue - 3108 for linking customer incidents where problems are proven to be related to this fault.  Typical problems involve not being able to send or receive email, and phones not registering.  You are still able to reach our web-site at www.gradwell.com

Update 06:50 - This issue appears to be resolved, however, we are awaiting an update from our network team to confirm the status of the resolution.

Update: This issue was resolved just before 07:00 and was related to one of our transit providers’ routers due to overnight maintenance.  If you are experiencing any issues with phones this morning, please try to reboot them, many phones will not continually retry if there is a network issue.  If problems persist, please contact our support team, who will be happy to advise.

This should result in a quicker, more stable service.

Update 13:47: The file-system issue with our primary control panel cannot be automatically repaired.  Our server team is continuing to work to try to bring this server back online, however, we are also working to provision a new machine to ensure we can restore control panel services, should the original control panels not be recoverable.

Update: 14:57: Hosting and VoIP control panels have been restored.  Logins have been tested.  However, we now need to ensure all partner portals are functioning and restore internal monitoring.  If customers continue to see any issues with the control panels, please submit an incident via our portal and we will look into the issue for you.  Again, we apologise for any inconvenience caused by this issue.

Update (10:40): We have now confirmed this appears to be a fault with our CRM platform, and we have raised a support ticket with the vendor.

Update (20:40): We have still not had any satisfactory response from our vendor.  We have however applied a workaround which seems to have fixed the immediate problem of control panel access failing, and will continue to monitor the situation, and work with our vendor for a full solution.

Update (23:40): Our vendor has now fixed this issue at their end and all services are working normally.

Update 17:00: We are currently experiencing a problem with inbound calls due to this issue, call setup may be affected.  We are working to bring back the affected VoIP SIP routers.

Update:17:15: This issue has now been resolved.  We have restarted several services which rely on the ‘Io’ back-end storage.  We apologise for any inconvenience caused.

This issue was resolved by our systems team Sat Oct 31 02:01:29 GMT and was related to a VMware fault, which we are following up with the vendor.