Update - This issue has now been resolved.

This issue has now been resolved and all services appear to be up and functioning normally with minimal interruption to service.

Please note this is a correction to our previously announced maintenance window tonight.  The correct date is Friday October 10th, apologies for any confusion caused.

Our back-end CRM system will be unavailable for maintenance Friday evening from 10PM.  This maintenance work will result in front-end functionality such as access to incidents and the VoIP control panels being partially or wholly unavailable during the maintenance.  Back-end services such as customer-hosted web-sites, crontabs and VoIP service will not be affected.

Please ensure any updates that are needed for your account to function during this time are completed before the maintenance window.  Thank you.

Update 11:10: This problem was resolved just before 10:00 and was caused by a disconnected private LAN connection affecting only th-shell-1.gradwell.net

Update 21:25: An on-site engineer in London has been authorised to manually restart one of the machines in our virtualised server cluster which will be done within the next few minutes to restore capacity to our network. We are currently aware of two potential issues with our mail file servers (3 & 7) and these will be addressed as soon as possible.

Update 22:20: We are currently experiencing an extended outage of one of the servers in our virtual infrastructure cluster.  The majority of services are functioning correctly.  However, we are aware that the following services are experiencing issues:

One of one load-balanced POP3 servers is currently offline which may result in intermittent POP3 collection problems.
Our back-end mail storage server, v-mail-file-3 is offline.  This will result in an outage for customers with mailboxes on this platform.  Mail will continue to be held on our inbound mail queues, however delivery will be delayed and this will result in errors collecting mail.  We apologise for this inconvenience and assure all customers that we are working with our engineers to ensure the service are restored as soon as possible.
Our UK SSL site is currently offline, again, we are working to restore this as soon as possible.

Update 22:50: We have raised an urgent support request with VMware and one of their engineers is now working to restore service.  When this is completed we will work with VMware to establish why the automatic resilience measures did not maintain a running service.

Update 23:10: We have now restored the majority of services, including uk-ssl.com. We are currently waiting for a full disk check to complete on v-mail-file-3 and are performing a maintenance restart on v-mail-file-7 which we expect to interrupt service only briefly.

Update 00:00: The disk checks on the 2 file servers have now completed and we believe all services are back to normal.

Update 12:35
We have restored database connectivity.  This issue was caused by a DNS issue internally which is currently being investigated in more details to ensure that more redundancy is added to ensure this does not happen again.

We are still seeing issues with whitelabel VoIP partners’ control panels which we are investigating currently.

Update 13:45
Our partner login areas are now working normally again, this issue is related to our database, which due to being unavailable had a backlog of work, which has now cleared.  We apologise for the inconvenience this will have caused you.

Due to a memory error, we have lost access to the management unit in our main cisco switch stack in Telehouse and therefore need to reboot one of the switch nodes.

Although the switches contain redundant controllers, we cannot be sure that the backup will take over, and there may be a short outage to our core network whilst it reboots.

As a precaution, we plan to shutdown most of our virtual servers to a bare minimum, and specifically we will switch off those providing the email system (as their hard disks are accessed via network based storage), and therefore no email collection or delivery will be available during this maintenance window.

We do expect our voip platform to continue working, although some calls may be dropped.

We expect this work to take under 15 minutes to complete, but please accept our apologies for this interuption to service.

Update - 03:15

The switch was successfully rebooted at 01:30 and nearly all services were fully functional by 02:00.  At the time of writing (03:15) we believe all services are back to normal.

Thank you for your patience during this maintenance and we apologize for any inconvenience caused.

Customers will receive an error message when trying to access either the Hosting or VOIP Control Panels, this outage is also affecting our partners.

We have engineers currently investigating this problem and will post an update once further information has been received.

This issue has now been resolved by our oncall engineering team, if you still experience problems accessing your Control Panel, please raise a support incident via the Gradwell Portal.

We are investigating the issue and will update gradwellstatus.com when we have further information.