Starts: 2010/03/14 22:00 Ends: 2010/03/14 22:30
On Sunday night we will upgrade the server handling VoIP presence information. There will be no visible functionality change - only bug fixes. No calls or registrations will be affected during the change, but phone statuses will not update for a couple of minutes during the upgrade.
Update (Friday, 12:50): Due to a number of problems with the presence server today, we have performed this maintenance earlier than planned and it is now complete. We will continue to monitor the presence server today.
From Sunday 14th March, the way we handle some call failures will change. Specifically, calls to extensions with no registered devices will be rejected more quickly with the correct code “404 Not found”, intead of previous “503″.
This change will have no effect on the majority of users, but it may affect a small number of users who have any automated call systems. Note that response codes on PSTN call failures will not change, so if you have automated call system which call PSTN numbers, this change will not affect you.
This change will not cause any downtime, since we are changing the way our call routing is generated, and no systems need to be taken down. Due to the way the routing is generated, it will not affect all customers immediately, but will be rolled out over the coming weeks.
Some users on ‘glacier’ are seeing issues collecting mail via POP3/IMAP this morning. Our system admin team are working on this now and we will update here as soon as further information is available.
We apologise for any problems this may be causing you.
***Update 13:55***
We have implemented a fix for the earlier mail problems and have rebalanced all mail stores. We also have further capacity coming online within the next two weeks to ensure these problems do not reoccur.
We apologise again for any problems caused by these earlier issues
We have had a notification from our DSL carrier that multiple LLU exchanges are showing as down around the Luton area. Affected customers will not have any services that run over DSL Their TMC are investigating and we will update when further information is made available. Please note, this will only affect LLU lines, IPstream and Datastream lines are unaffected
***Update 16:40***
Our DSL carrier has advised these faults are now totally cleared. We apologise to any of our DSL users that were affected.
We are currently experiencing a problem with one of our VMware hosts. To return full functionality we need to reboot one of our blades. This will cause an interruption to several services including shell.gradwell.net, some websites hosted on ‘dixie’ and hosted exchange services.
We will post here again upon completion of this emergency work. We apologise in advance for any problems caused by this.
***Update 13:54***
We have fully powered off the problem blade and are restarting services now. Some virtual machines will require a filesystem check so these will take slightly longer to complete.
***Update 14:12***
We have now restored all services. If you are still seeing any problems, please do raise an incident with our support team and we will investigate on a case by case basis. Our apologies again for the loss of service.
We will be installing a new MySQL 5 database server this week and this will unfortunately cause some downtime whilst the migration is underway. This maintenance window will be Wednesday 10th March from 20:00 to 22:00
No user intervention will be needed as all current SQL databases will be migrated by our system admin team during the above window. Only users with databases being served from mysql5db-1.gradwell.net will be offline, all other database users will be unaffected.
If you are unsure of where your SQL databases are, please log in to your hosting control panel and click on ‘Manage Databases’ This will list all of your current databases and the server on which they are located. We apologise in advance for any problems caused by this downtime.
***Update 21:20***
Due to the amount of time taking to dump the necessary databases, and double check the dumps contain all the data, it would appear we will need more time than initially expected. Therefore we are unable to complete the works tonight. We will reschedule this maintenance for Friday night and will post here tomorrow with an updated window.
***Update 11:50 12th March***
We are now rescheduling the DB server installation for Wednesday 17th March between 21:00 to midnight. This will allow the sysadmin team more time to successfully complete the full SQL migration. We apologise for this change of date.
Over the last week we have identified a number of reports of customers getting ‘500′ errors when using our control panels, particularly when editing support tickets.
This is a problem with the way the control panel software has interacted with our database and we deployed a fix for this on Friday afternoon (5th March).
The problem is now resolved. Please accept our apologies for the problem and inconvenience this caused and thank you to the customers who aided in the diagnostic.
Please be aware we are currently seeing an outage one of our back end systems. Our system admins are currently looking the issue and we are trying to get this back up as soon as possible.
This will be affecting some of our customers websites and home directories while we attempt to restore full service again.
***Update 14:24***
This issue was cleared in full yesterday shortly after posting the initial status update. We have been closely monitoring the storage since then and have seen no further errors.
We are now happy to close this status notification and we apologise for any problems this might have caused you.
Starts: 2010/03/08 00:01 Ends: 2010/03/08 01:00
At midnight on Sunday night/Monday morning we are performing some updates on our single/multi-user VoIP platform. We anticipate approximately 10 minutes of downtime. During this period, users will be unable to make or receive VoIP calls, and extensions may fail to register. Any calls in progress when the maintenance start will not be dropped.
The following VoIP services will not be affected by this maintenance:
- All trunks (SIP and IAX, inbound and outbound)
- Remote Call Forwarding
- Fax-to-Email
- Diverts on single/multi-user VoIP extension (extensions will not receive calls, but diverts will still function) - provided the divert is configured through our control panel
- Hosted Unified Comms
Please accept our apologies for any inconvenience this may cause, and if you have any questions, please contact our support team.
Update (00:48): This maintenance work has now completed, with full service restored at 00:40.
Several LLU exchanges connected to the Shareshill aggregation router have lost connectivity due to a backhaul fault, customers connected to these exchanges will not have any DSL connection available.
All exchanges will still have 999 emergency call functionality.
BT are currently investigating. Exchanges affected have been listed below.
Broadwell, Great Barr, Stourbridge, Kingswinford, Kidderminster, Tipton and Dudley.
UPDATE 13:34 - We have been informed that BT engineers are onsite carrying out fibre repairs , further updates will follow.
UPDATE 15:09 - BT Engineers are continuing to repair the fibre that has become damaged.
UPDATE 09:12 25/2/10 - Services have now been restored to all exchanges above apart from Tipton, BT engineering are continuing to investigate and repair fibre.
UPDATE 09:02 26/2/10 - Services to all exchanges listed above have now been restored.
