Gradwell Service News

We will be switching off parts of our core network in the very early hours of Friday morning to perform some upgrades and to increase power resiliency. Some parts of our network will be unavailable for short periods of time between 00:00 and 05:00. During this period all phones will re-register at some point, some websites will be offline and all Gradwell DSL lines will lose connection for up to 20 minutes.

Whilst the whole network will be ‘at risk’ for the entire period, the main windows for major network outages are specific per device. Our work plan will be as follows:-

Core Switching - 01:00-2:00

ADSL Routers - 02:00-02:30

Border Routers - 02:30-03:30

Peering Routers - 03:30-04:00

We apologise in advance for any loss of services caused by this and if you experience any problems after this work has been completed, please contact support@gradwell.com

*** UPDATE 07:05 ***

We are investigating a problem with the web cluster following this maintenance.  Status will be updated here as soon as further information is available.  We apologise for any inconvenience caused.

***Update 13:38***

We have been monitoring all upgrades and they are performing as expected. All work was successfully completed. We apologise again for any loss of service during these upgrades.

We will be migrating our Mailman ‘list.yourdomain.com’ interface to a server in a different datacentre shortly, and we need to change the IP address of the system from 193.84.87.101 to 212.11.71.211.  We will be automatically modifying all hosted domains automatically, and the change should take effect in approximately an hour.  Due to the nature of list archives and mail queues, we will not be able to operate both servers during the switch over, so customers may experience some, minimal down time with this service.

Update 17:11 - We have resolved an issue with the new server configuration which was preventing the migration.  The old Mailman server has now been disabled, and the web interfaced removed to ensure nothing changes before our final migration to the new server.

Update 19:47 - Apologies for the delay with this new server, which had to copy the remaining mail over from the old server, without any changes being accepted, the new server is now active on 212.11.71.211, all hosted DNS has been updated and is now propagating over the internet.  This maintenance has been essential because the IP address space available to us in Sovereign House is being re-allocated.

Our senior engineers are currently investigating an issue with our Mailman implementation which is currently causing delivery issues, although the web interface is now working correctly.

Update 22:30 - Our Mailman migration is now complete, our engineers have found and corrected the issue with mail not being delivered and the server has been processing queued mail correctly for the past 30 minutes.

We will be conducting overnight maintenance in our London datacentre this evening which will involve moving our database hosts and removing our legacy FreeBSD web cluster servers.  The maintenance will commence at 22:00 and is expected to continue until 06:00.  During this time, some services may not be available for short periods.  We will be decommissioning the legacy servers - Lilac and Ochre, SIP and shell services respectively, and mysql5db-1 will be temporarily unavailable, as it will be physically re-located within the datacentre.  Our core hosting database will also be moved, which will affect most services, including the ability to log into mailboxes.  We aim to keep this downtime to a minimum.

Update 02:45: Please note that we are now performing database maintenace, which will prevent VoIP calls completeing, or mail from being collected.  Websites will continue to function, however, configuration changes cannot be completed at this site and control panels will bee temporarily inaccessible.

Update: 02:55: We have moved our core hosting database to a temporary location to minimize any disruption.  All services should now be available again.

Update 03:23 - We have taken down mysql5db-1 (customer MySQL server) and one core hosting database slave. Inbound and outbound calls plus mail is 100% operational, some customer databases will be unavailable for a short duration. Web hosting should also be fine.

Update: This maintenance was completed by 08:00.

This evening from approximately 23:00 until 00:00, we will be performing maintanance on our web load-balancers, which is part of the work for removing the legacy FreeBSD cluster.  There is a possibiity of a brief outage on all web clusters as this work progresses.  We will update this site once the maintenance has been completed, or with any further updates.  Thank for you for patience during these essential works.

This maintenance has now been completed, all PHP 5 sites are now running from our new virtualised PHP 5.2 platform.  Our legacy load balancer has been decommissioned.

We are performing scheduled maintenance on 16th September 2009 which
will involve a period of outage across all products and services.

During this maintenance window we will be:

• performing software upgrades to core network equipment
• performing software upgrades to DSL network equipment
• running additional cabling for the core network to ensure
  tolerance of any important link failure, and to add
  pro-actively add capacity to areas of the network which
  will experience significantly increased throughput over
  the coming months as Gradwell continues to grow
• updating configuration on all aspects of the network to
  fully support native IPv6 addressing
• upgrading switches used with our SAN solution to improve
  storage reliability with the vSphere ESX 4.0 platform
• upgrading head nodes of the ESX storage solution to provide
  increased reliability, throughput and manageability
• running additional cabling for the core network to ensure
  tolerance of any important link failure

Outage will begin on 16/09 at 03:00 (GMT) and we expect to be completed
within 60 - 90 minutes; whilst the service may come up again quite
quickly it should be considered “at risk” between 03:00 and 05:00.

Update: This work was completed with a minimal window of downtime, lasting about ten minutes.

This announcement is intended to serve as an End of Life (EOL) notice
against two of our legacy shell servers, which are running FreeBSD.
Removal from service will occur on 30th April 2009 and all customers are
hereby advised to migrate to ’shell.gradwell.net’ in the next 4-5 weeks.

End of Life
30th April 2009

Affected Hosts:
ochre.gradwell.net
newred.gradwell.net

Any scheduled jobs (via cron) or screen sessions running on these hosts
must be migrated over prior to the removal date, customers should note
that the new shell servers are running CentOS 5.x and not FreeBSD.

Any data which is not located in your home directory on these servers
will *not* be migrated, so please ensure you do not use other locations
on the filesystem (ie: /tmp) for essential data.

Please contact the support team if you have questions.

We are aware of an issue with our back-end systems including PHP clusters, affecting live sites and test domains, and are working hard to resolve the problems as quickly as possible. All members of our technical team are currently dedicated to resolving this problem for customers as quickly as possible and drawing up contingency plans for getting services back up and running as quickly as possible.  We will be updating customers with a full report as soon as possible.

Update 21 Jan 09 6:30am: All websites and domains were brought back online overnight.  If you are still experiencing problems, please contact Gradwell Customer Services.

We are currently investigating a problem with one of our virtual server hosts which may be affecting some network services including customers on v-mail-file-8, lon-pbx-6 and 14. We will customer customers with more details as soon as they are available.

09:31 This issue has been resolved, although our server team will be working on the network throughout the day to ensure services are stabilised as soon as possible.

We are currently aware of an issue which is affecting our virtualised network in London.  Our server team are investigating and we will update customers as soon as possible with further information.

Update 14:30 Our server team are currently restarting some of the key servers within our Telehouse cluster and we are working closely with our platform vendors to identify the cause of the network issue we are currently seeing.  We hope to provide a resolution to this problem as soon as possible.

Update 15:00 We are currently recovering all affected services and expect all services to be fully online shortly.

Update 15:40 Whilst the majority of services are now back in operation, including PHP 4.4 and 5.2 services, and our VoIP NAT proxies, we are experiencing some issues with some of our email services, which our administrators are working to bring back online as soon as possible.  Our virtual PBX lon-pbx-14 is not yet back in operation.  We apologise for the ongoing inconvenience to customers using these services.

Update 17:35 Our administrators have brought out IMAP,POP3 and SMTP services back online, however, v-mail-file-1,5,6,7,8,9 remain offline and we are working to bring them back online as soon as possible.  Customers using these file servers may encounter errors when attempting to send email, however, this will be due to the email client saving the email to the remote folder, the email itself will be sent out.

Update 18:10 Our VoIP Centrex server lon-pbx-14 has now been brought back online and is handling calls correctly.  We apologise for the length of time this server was unavailable.

Update 19:30 All known issues with our servers have now been resolved.  If you are experiencing any remaining problems, please contact our customer services team, who will be able to advise you and inform our server team of any problems which may remain.  Apologies for the extended problems today.  Our server team are working through a lot of changes which should make our network more resilient in future.

Starts: 2009/01/14 22:00 Ends: 2009/01/15 02:00

Overnight Wednesday / Thursday this week, we will be rotating servers one by one out of our VMWare ESX cluster for essential maintenance.  We are adding an additional 4 CPU cores per server, to increase the overall capacity of the ESX cluster.

During this time, our services will continue to run, but there may be small amounts of disruption.

We apologise for the short notice for this change.