(39) (9) (64) (63) (63) (10) (12) (3) (108)We have had a notification from our DSL carrier that multiple LLU exchanges are showing as down around the Luton area. Affected customers will not have any services that run over DSL Their TMC are investigating and we will update when further information is made available. Please note, this will only affect LLU lines, IPstream and Datastream lines are unaffected
We are currently experiencing a problem with one of our VMware hosts. To return full functionality we need to reboot one of our blades. This will cause an interruption to several services including shell.gradwell.net, some websites hosted on ‘dixie’ and hosted exchange services.
We will post here again upon completion of this emergency work. We apologise in advance for any problems caused by this.
***Update 13:54***
We have fully powered off the problem blade and are restarting services now. Some virtual machines will require a filesystem check so these will take slightly longer to complete.
***Update 14:12***
We have now restored all services. If you are still seeing any problems, please do raise an incident with our support team and we will investigate on a case by case basis. Our apologies again for the loss of service.
We will be installing a new MySQL 5 database server this week and this will unfortunately cause some downtime whilst the migration is underway. This maintenance window will be Wednesday 10th March from 20:00 to 22:00
No user intervention will be needed as all current SQL databases will be migrated by our system admin team during the above window. Only users with databases being served from mysql5db-1.gradwell.net will be offline, all other database users will be unaffected.
If you are unsure of where your SQL databases are, please log in to your hosting control panel and click on ‘Manage Databases’ This will list all of your current databases and the server on which they are located. We apologise in advance for any problems caused by this downtime.
Over the last week we have identified a number of reports of customers getting ‘500′ errors when using our control panels, particularly when editing support tickets.
This is a problem with the way the control panel software has interacted with our database and we deployed a fix for this on Friday afternoon (5th March).
The problem is now resolved. Please accept our apologies for the problem and inconvenience this caused and thank you to the customers who aided in the diagnostic.
Please be aware we are currently seeing an outage one of our back end systems. Our system admins are currently looking the issue and we are trying to get this back up as soon as possible.
This will be affecting some of our customers websites and home directories while we attempt to restore full service again.
***Update 14:24***
This issue was cleared in full yesterday shortly after posting the initial status update. We have been closely monitoring the storage since then and have seen no further errors.
We are now happy to close this status notification and we apologise for any problems this might have caused you.
Starts: 2010/03/08 00:01 Ends: 2010/03/08 01:00
At midnight on Sunday night/Monday morning we are performing some updates on our single/multi-user VoIP platform. We anticipate approximately 10 minutes of downtime. During this period, users will be unable to make or receive VoIP calls, and extensions may fail to register. Any calls in progress when the maintenance start will not be dropped.
The following VoIP services will not be affected by this maintenance:
Please accept our apologies for any inconvenience this may cause, and if you have any questions, please contact our support team.
Update (00:48): This maintenance work has now completed, with full service restored at 00:40.
Several LLU exchanges connected to the Shareshill aggregation router have lost connectivity due to a backhaul fault, customers connected to these exchanges will not have any DSL connection available.
All exchanges will still have 999 emergency call functionality.
BT are currently investigating. Exchanges affected have been listed below.
Broadwell, Great Barr, Stourbridge, Kingswinford, Kidderminster, Tipton and Dudley.
UPDATE 13:34 - We have been informed that BT engineers are onsite carrying out fibre repairs , further updates will follow.
UPDATE 15:09 - BT Engineers are continuing to repair the fibre that has become damaged.
UPDATE 09:12 25/2/10 - Services have now been restored to all exchanges above apart from Tipton, BT engineering are continuing to investigate and repair fibre.
UPDATE 09:02 26/2/10 - Services to all exchanges listed above have now been restored.
We have discovered that Microsoft Hotmail/Yahoo/Live networks have decided to block our mailservers from sending outbound mail directly to their MX servers. We have requested removal from this blocklist and are awaiting Microsoft to action.
We apologise for any inconvenience this causes and would like to remind anyone using us to forward mail to one of these accounts to not ‘report as spam’ if the mail is only forwarded through us. This blacklists our network, not the originating network.
***Update 10:10***
We are now only subject to the normal Microsoft ‘throttling’ for all of their inbound servers so the vast majority of emails should now be reaching their intended target. We apologise for any problems resulting from this and we have asked for our sending limits to be raised across all Microsoft servers.
We are investigating a problem with the web cluster following last night’s maintenance. Status will be updated here as soon as further information is available. We apologise for any inconvenience caused.
*** Update 07:50 ***
This problem was inadvertently caused by some emergency security changes made last night. Those changes have now been reverted, pending review and the web cluster has been re-synchronised. All PHP 4 and 5.2 sites are now back online.
We will be switching off parts of our core network in the very early hours of Friday morning to perform some upgrades and to increase power resiliency. Some parts of our network will be unavailable for short periods of time between 00:00 and 05:00. During this period all phones will re-register at some point, some websites will be offline and all Gradwell DSL lines will lose connection for up to 20 minutes.
Whilst the whole network will be ‘at risk’ for the entire period, the main windows for major network outages are specific per device. Our work plan will be as follows:-
Core Switching - 01:00-2:00
ADSL Routers - 02:00-02:30
Border Routers - 02:30-03:30
Peering Routers - 03:30-04:00
We apologise in advance for any loss of services caused by this and if you experience any problems after this work has been completed, please contact support@gradwell.com
*** UPDATE 07:05 ***
We are investigating a problem with the web cluster following this maintenance. Status will be updated here as soon as further information is available. We apologise for any inconvenience caused.
***Update 13:38***
We have been monitoring all upgrades and they are performing as expected. All work was successfully completed. We apologise again for any loss of service during these upgrades.
