Gradwell Service News » Hosting

RESOLVED: Web sites

David Palmer — Fri, 19 Feb 2010 07:07:40 +0000

We are investigating a problem with the web cluster following last night’s maintenance. Status will be updated here as soon as further information is available. We apologise for any inconvenience caused.

*** Update 07:50 ***
This problem was inadvertently caused by some emergency security changes made last night. Those changes have now been reverted, pending review and the web cluster has been re-synchronised. All PHP 4 and 5.2 sites are now back online.

RESOLVED - Home File Server Outage

paul — Mon, 11 Jan 2010 08:42:53 +0000

We are currently experiencing an outage of one of our home file servers. We are attempting to restore service as quickly as possible. Services affected include web hosting, shell servers, ftp servers and any other service that relies on /home folders.

***UPDATE 10:36***

We have temporarily moved the affected user filestores to a different virtual machine and these user folders should now be available but only in read only mode. Our sysadmin team are currently working on rebooting and reinstating the original virtual machine to enable full access.

***UPDATE 12:30***

We have moved all filestores back to the original host and have been monitoring for the last 30 minutes. All is running as expected and service should now be back to normal.

If you are still seeing any issues, please raise a support incident to support@gradwell.com and we will investigate on a case-by-case basis.

Our apologies for any problems this has caused you.

RESOLVED - MYSQL5 database server error messages

Dean Green — Sat, 26 Dec 2009 19:54:31 +0000

Some users who have databases on our MYSQL5 database server are seeing some errors at present. Our on-call system admin engineer has been alerted and is looking into what is causing these messages.

We will update here ASAP and again when this is fully rectified. We apologise for any problems this is causing you.

***Update 21:00***

This has now been fully cleared. The messages were caused by one user having some bad code which was using all available connections to the SQL5 database server. We have now enforced hard limits on this user to stop this happening in the future.

RESOLVED: Control panel access

David Palmer — Mon, 21 Dec 2009 13:01:17 +0000

We are currently investigating a file-system problem which is preventing some of our control panels from functioning correctly. Apologies for any inconvenience caused. We aim to have this resolved very shortly.

Update 13:47: The file-system issue with our primary control panel cannot be automatically repaired. Our server team is continuing to work to try to bring this server back online, however, we are also working to provision a new machine to ensure we can restore control panel services, should the original control panels not be recoverable.

Update: 14:57: Hosting and VoIP control panels have been restored. Logins have been tested. However, we now need to ensure all partner portals are functioning and restore internal monitoring. If customers continue to see any issues with the control panels, please submit an incident via our portal and we will look into the issue for you. Again, we apologise for any inconvenience caused by this issue.

COMPLETED: Maintenance on dixie file server

David Palmer — Tue, 24 Nov 2009 12:04:03 +0000

Starts: 2009/11/26 23:00 Sends: 2009/11/27 00:00

We will be performing some maintenance on the ‘Dixie’ file server which serves approximately one quarter of our customer home directory storage, serving web sites and back-end regular ‘cron’ jobs. The maintenance will be completed from 23:00 until mid night, Thursday 26th November 2009. This will involve a brief outage for web-sites whilst we correct an NFS consistency problem across the network.

Update 23:00: This maintenance window is now in progress.

Update 00:08: We completed these changes at 23:40 and have checked that services are working correctly. Should you experience any remaining issues, please submit a support incident or call out an engineer if urgent. Thank you.

Control Panel And IAX Proxy Maintenance

Stuart Herbert — Tue, 03 Nov 2009 22:02:36 +0000

Starts: 2009/11/03 22:00 Ends 2009/11/03 23:59

We are performing maintenance on the following systems and services tonight:

VoIP Control Panel (new product launch)
Hosting Control Panel (misc server maintenance)
IAX Proxy (iax.trunk.gradwell.com) (fix for audio-out issues)

We apologise in advance, but during this maintenance, these systems/services may not be available to our customers. We will do our best to keep disruption to a minimum.

UPDATE 22:38: Maintenance has been successfully completed.

RESOLVED: DNS zone file changes

David Palmer — Fri, 30 Oct 2009 16:30:10 +0000

Our server team are currently investigating an issue with our DNS update system which updates customer zone files after control panel changes. Please note that this issue also affects secondary DNS. Our servers will continue to answer with older zone files, however any chances will not be honoured. We apologise for any inconvenience this will cause. We are currently following this issue up with our virtualisation vendor.

This issue was resolved by our systems team Sat Oct 31 02:01:29 GMT and was related to a VMware fault, which we are following up with the vendor.

RESOLVED - Intermittent webserver issues

Dean Green — Tue, 20 Oct 2009 08:24:30 +0000

We were seeing some intermittent problems across both the PHP 4.4 and PHP 5.2 web clusters. Our system admin team have cleared this issue but we are still monitoring these servers.

We will update here when we are satifised that both clusters are serving at 100%. We apologise for any problems this has caused you.

***UPDATE - 10:15***

We are now satisfied that the web clusters are serving all content as expected and are happy to resolve this status post. If you are still seeing any problems, please raise a support incident to support@gradwell.com

COMPLETED: Planned database migrations

paul — Wed, 14 Oct 2009 18:06:22 +0000

We will be taking down one of our database servers (white.gradwell.net) in 3 hours at 2200BST to perform migrations to a new server. During the migration process all databases on white.gradwell.net will be unavailable. Customers affected have already been notified by email, if you have not already received an email then your service will not be affected.

** UPDATE ** new server is online as of 23:20BST and serving databases

RESOLVED: Customer crons

David Palmer — Wed, 23 Sep 2009 17:22:58 +0000

Many customer crontabs have not been run between 11:00 and 18:00 today due to emergency configuration changes which were intended to control load on our main shell server. We have now restored this functionality and the crontab command will give details of the server running the crontabs. Please note that customers may need to manually run any jobs which were scheduled to be started within these hours, if they are important. Apologies for any inconvenience.