Currently we are experiencing a problem which will affect customers who use our outbound NAT proxy (nat.gradwell.net).
Our oncall engineers have been alerted to the problem and we are awaiting further updates.
If you are a customer experiencing this problem, you will find you are unable to make outbound calls and your phone’s LCD screen may display an error such as “1/SL Proxy Authentication Required”.
*** UPDATE 14:33 ***
This issue has now been resolved and calls are completing successfully through our network.
Some users are currently seeing extensions dropping registration and unable to reregister. Our on-call system admin team are looking into this now. We will post any updates here as soon as possible.
We apologise for any problems this is causing
***UPDATE*** 11:30
This has now been fully cleared and all extensions that were showing offline are now registered. If you are still seeing problems after a reboot of your VoIP device, please raise a support incident to support@gradwell.com Our apologies again for any problems this has caused you.
Some users who have databases on our MYSQL5 database server are seeing some errors at present. Our on-call system admin engineer has been alerted and is looking into what is causing these messages.
We will update here ASAP and again when this is fully rectified. We apologise for any problems this is causing you.
***Update 21:00***
This has now been fully cleared. The messages were caused by one user having some bad code which was using all available connections to the SQL5 database server. We have now enforced hard limits on this user to stop this happening in the future.
Gradwell Customer Service will be available for all customers over Christmas and New Year, but there will be changes to our hours of operation.
From Monday 21st to Christmas Eve, telephone and email support will be provided from 9am to 5pm. On Christmas Day there will be no support. Limited email support is then provided on Saturday 26th and Sunday 27th. Bank Holiday Monday (28th) will have telephone and email support from 9am to 1pm.
9am to 5pm services then resumes from Tuesday 29th to Thursday 31st. Telephone and email support is available on New Years Day from 9am to 1pm. Regular weekend support hours are then provided 2nd and 3rd January.
Normal support hours of 8am to 8pm then resume on Monday 4th.
At a glance:
* Monday 21st: Telephone and email support, 9am – 5pm
* Tuesday 22nd: Telephone and email support, 9am – 5pm
* Wednesday 23rd: Telephone and email support, 9am – 5pm
* Christmas Eve: Telephone and email support, 9am – 5pm
* Christmas Day: CLOSED
* Saturday 26th: Limited email support
* Sunday 27th: Limited email support
* Monday 28th: Telephone and email support, 9am – 1pm
* Tuesday 29th: Telephone and email support, 9am – 5pm
* Wednesday 30th: Telephone and email support, 9am – 5pm
* New Year’s Eve: Telephone and email support, 9am – 5pm
* New Year’s Day: Telephone and email support, 9am – 1pm
* Saturday 2nd: Telephone and email support, 9am – 1pm
* Sunday 3rd: Limited email support
* Monday 4th: Telephone and email support, 8am – 8pm
December 24th, 2009, 11:06 am by Thomas Causer News
We will re restarting our control panel systems at 17:30 this evening. There is likely to be a brief period when the control panels are unavailable. This is necessary due to some software maintenance to resolve a performance problem, and to resolve a SCSI issue with a new drive we are adding to hold logging information.
Update 17:16: This maintenance has been cancelled. A problem in our web-site was causing very high load and has been corrected. We will no longer require this maintenance window.
We are currently investigating a file-system problem which is preventing some of our control panels from functioning correctly. Apologies for any inconvenience caused. We aim to have this resolved very shortly.
Update 13:47: The file-system issue with our primary control panel cannot be automatically repaired. Our server team is continuing to work to try to bring this server back online, however, we are also working to provision a new machine to ensure we can restore control panel services, should the original control panels not be recoverable.
Update: 14:57: Hosting and VoIP control panels have been restored. Logins have been tested. However, we now need to ensure all partner portals are functioning and restore internal monitoring. If customers continue to see any issues with the control panels, please submit an incident via our portal and we will look into the issue for you. Again, we apologise for any inconvenience caused by this issue.
We are currently experiencing an issue with our backend CRM that is preventing customers from using our control panels. Customers may recieve an error “Unable to connect to backend CRM database” when attempting to login to or use our control panels. This issue has been escalated to our engineering team.
Update (10:40): We have now confirmed this appears to be a fault with our CRM platform, and we have raised a support ticket with the vendor.
Update (20:40): We have still not had any satisfactory response from our vendor. We have however applied a workaround which seems to have fixed the immediate problem of control panel access failing, and will continue to monitor the situation, and work with our vendor for a full solution.
Update (23:40): Our vendor has now fixed this issue at their end and all services are working normally.
We wish to advise customers that on Thursday 17th December, support hours will be from 08:00 until 17:00.
Our messaging service will continue to relay messages to our duty support technician, and we will natually respond to all service affecting issues.
Normal support will resume on Friday 18th. Further details will be posted regarding the support rota over the entire holiday period shortly.
We are seeing some mail delays at present. This was caused by a backlog caused by spam and our system admin team are clearing mail queues now.
We apologise for any problems this is causing you.
***UPDATE 11:05***
We have cleared all spam mail from our mail cluster and our mail queues are now returning to normal
Our NAT proxy, lon-ppc-3 which is used mainly by our partners has stopped responding and has been power cycled by our systems team. This will have resulted in any calls passing through it at the time being disconnected. We apologise for any inconvenience this will have caused. If customers see any further problems, we advise restarting VoIP phones before contacting our support team, who will be happy to assist.
