*** Update 07:50 ***
This problem was inadvertently caused by some emergency security changes made last night.  Those changes have now been reverted, pending review and the web cluster has been re-synchronised.  All PHP 4 and 5.2 sites are now back online.

***UPDATE 10:36***

We have temporarily moved the affected user filestores to a different virtual machine and these user folders should now be available but only in read only mode. Our sysadmin team are currently working on rebooting and reinstating the original virtual machine to enable full access.

***UPDATE 12:30***

We have moved all filestores back to the original host and have been monitoring for the last 30 minutes. All is running as expected and service should now be back to normal.

If you are still seeing any issues, please raise a support incident to support@gradwell.com and we will investigate on a case-by-case basis.

Our apologies for any problems this has caused you.

We will update here ASAP and again when this is fully rectified. We apologise for any problems this is causing you.

***Update 21:00***

This has now been fully cleared. The messages were caused by one user having some bad code which was using all available connections to the SQL5 database server. We have now enforced hard limits on this user to stop this happening in the future.

Update 13:47: The file-system issue with our primary control panel cannot be automatically repaired.  Our server team is continuing to work to try to bring this server back online, however, we are also working to provision a new machine to ensure we can restore control panel services, should the original control panels not be recoverable.

Update: 14:57: Hosting and VoIP control panels have been restored.  Logins have been tested.  However, we now need to ensure all partner portals are functioning and restore internal monitoring.  If customers continue to see any issues with the control panels, please submit an incident via our portal and we will look into the issue for you.  Again, we apologise for any inconvenience caused by this issue.

We will be performing some maintenance on the ‘Dixie’ file server which serves approximately one quarter of our customer home directory storage, serving web sites and back-end regular ‘cron’ jobs. The maintenance will be completed from 23:00 until mid night, Thursday 26th November 2009. This will involve a brief outage for web-sites whilst we correct an NFS consistency problem across the network.

Update 23:00: This maintenance window is now in progress.

Update 00:08: We completed these changes at 23:40 and have checked that services are working correctly.  Should you experience any remaining issues, please submit a support incident or call out an engineer if urgent.  Thank you.

This issue was resolved by our systems team Sat Oct 31 02:01:29 GMT and was related to a VMware fault, which we are following up with the vendor.

We will update here when we are satifised that both clusters are serving at 100%. We apologise for any problems this has caused you.

***UPDATE - 10:15***

We are now satisfied that the web clusters are serving all content as expected and are happy to resolve this status post. If you are still seeing any problems, please raise a support incident to support@gradwell.com

Web-sites on this server will be unavailable temporarily.  We apologise for any inconvenience caused, we aim to restore service within an hour.

Update 00:54: This issue has now been resolved and Dixie has been brought back online.  Web-sites may be slow for a few minutes as queued accesses are processed by the server.

**Update** 9:15

We are seeing further problems with the filesystem on Dixie this morning. Our server admin team are working on this now and we will post an update as soon as possible.

We apologise for any problems these filesystem issues are causing.

**Update** 12:25

To get Dixie back to stable working levels our server admin team need to remove it from active service to perform some block level disk checking. This will cause a disruption for any customer who is serving a website or is accessing file(s) within their home filespace if located on Dixie. We apologise for the continuing interruption and we will post an update here as soon as possible.

Update 15:05: Due to intermittancy of the home file server; Dixie, our server team will be restoring the file system in a read-only state.  This will allow the majority of sites to work normally until the service can be fully restored on an alternative server.

Update 20:05: Backups for fiel storage on Dixie have been recovered, and we are now recovering available data from the last 24 hours from Dixie.  We will shortly be switching over to the new server.

Update 13:06: This issue was resolved at midnight, however, shell and our legacy cluster could not reach the new server, this was resolved before 10:00 today.

Update 16:35 The Badlands server was brought back onine after a hardware upgrade at 16:10 and is now processing a backlog of emails. We have tested customer mailbox access and this is working correctly. If you are continuing to experience issues which are not related to delayed mail, please contact our customer services team.

Update 17:42: We are currently experiencing issues with our PHP 4.4 web load balancers and continuing to experience a back log of mail which cannot be processed. Our server team are currently upgrading hardware to ensure that more resources are available and will continue to update customers via this website.

Update 20:55: Our server team are still investigating this issue. As an emergency measure, some load intensive services such as AV scanning have been switched off. We are aiming to restore these services as quickly as possible, however, the performance of mail and web in general will need to be resolve first. We apologise for the ongoing inconvenience.

Update 21:25: Our technicians have now diagnosed and resolved a networking configuration issue which was impacting on the performance of our back-end storage SANs, causing our mail and web systems to have difficulty serving requests and handling mail in a timely fashion.

We are monitoring traffic and will restore AV services as soon as we confirm all queues are cleared.

Update Sat 25th July 11:00: We have made an emergency fix to flush outbound mail queues, and these queues have now been cleared, we are aware that an issue remains with our core network and are awaiting a fix as soon as possible.  Further details will follow as soon as they are available.  At this time we believe all customer-facing issues are resolved, however, should you be encountering an issue, please submit an incident via our support portal or email support@gradwell.com

Update 16:41: Full Anti-virus and spam scanning services have now been restored to the network, our server and network teams will be working next week to re-balance load and set up additional network monitoring to provide faster answers should an issue like this arise in future.