Over the last week we have identified a number of reports of customers getting ‘500′ errors when using our control panels, particularly when editing support tickets.
This is a problem with the way the control panel software has interacted with our database and we deployed a fix for this on Friday afternoon (5th March).
The problem is now resolved. Please accept our apologies for the problem and inconvenience this caused and thank you to the customers who aided in the diagnostic.
Our engineering team are currently investigating an issue with our web portal, we apologise for any inconvenience caused. Access to our VoIP and hosting control panels are not affected.
Update: 18:15: This issue has now been resolved. A new revision of the portal was due for Sunday, and in order to minimise downtime, this has been brought forward. If customers experience any problems with the new portal, please raise an incident with our support team. Most functionality is also available in our VoIP and hosting control panels, with the exception of broadband controls.
We are currently investigating a file-system problem which is preventing some of our control panels from functioning correctly. Apologies for any inconvenience caused. We aim to have this resolved very shortly.
Update 13:47: The file-system issue with our primary control panel cannot be automatically repaired. Our server team is continuing to work to try to bring this server back online, however, we are also working to provision a new machine to ensure we can restore control panel services, should the original control panels not be recoverable.
Update: 14:57: Hosting and VoIP control panels have been restored. Logins have been tested. However, we now need to ensure all partner portals are functioning and restore internal monitoring. If customers continue to see any issues with the control panels, please submit an incident via our portal and we will look into the issue for you. Again, we apologise for any inconvenience caused by this issue.
Some users are seeing duplicates of the same zone file within the hosting control panel.
If you are affected by this, we would ask you to not make any changes until we have resolved this as any change made may possibly corrupt the zone file.We will post an update here as soon as possible.
We apologise for any problems this causes you.
****Update 10:26****
This issue is now totally cleared. If you are seeing any oddities or missing any zone files, please raise an incident with support and we will investigate and restore.
We experienced a power issue with one of the server hosts which runs several important services such as lon-pbx-1 (VoIP) and our customer control panels at 22:20. Although the machine recovered quickly these services could not be made to work as quickly as we hoped on our next generation of servers and so have been rolled back as an emergency measure. We apologise for any inconvenience caused by this. The problem was resolved and all services brought back online by 23:30. Our backup server for VoIP operated for lon-pbx-1 during this time. Customers may need to restart their VoIP equipment to ensure they are using the primary server again.
We are currently experiencing a technical issue where access to the Gradwell Hosting and VoIP control panels may be unavailable for customers.
Customers upon trying to login to their control panels will be seeing a System Error message displayed in their web browser.
An oncall engineer has been altered to this problem and we will update this status posting once further information is received.
Update 08:30: An issue has been investigated and fixed with our internal API server which has been restarted and resolves this problem.
We have identified a DNS issue which affects access to our core database services. This issue will affect call set up times and access to our control panels. Our server team are currently investigating this problem and will update gradwellstatus as soon as possible.
This issue has now been resolved, due to some configuration changes on our core database server cluster.
We are currently experiencing an outage of one of our core storage servers - Europa which will temporarily affect access to our part of our web-services and some other services. We will publish more information as soon as it is available.
Update 16:25: Our core storage server has now come back online, however, some of the dependent services, such as webmail, some clustered SMTP outbound and web servers have some file system errors are being restarted.
Individual customer mailboxes or sites are not affected due to our migration to HP/Lefthand storage.
RESOLVED by 16:51: We have now restarted any affected machines, core VoIP services were not affected, although we have had to restart both of our inbound Asterisk-based fax servers. The majority of customers are not using these. Our web load balancer was briefly unavailable, which will have caused a brief outage with customer hosted web-sites. Our monitoring system is continuing to report some machines which have had file system errors and we are intervening as appropriate, however, the majority of these are clustered services, and so customers should not see any effects.
We have made the following changes to the Gradwell control panels this afternoon:
- All of our control panels are now hosted on *.gradwell.com.
- Our control panels share a new look & feel and Gradwell’s new logo.
These changes are the first step in the launch of Gradwell’s refreshed brand. There will be more changes from us in the coming days.
If you spot any problems with any of our control panels, please contact Customer Services to let us know.
We are presently experiencing a problem with our control panels which will prevent customer logins. This is due to a problem with our CRM provider’s back-end which we are currently investigating. All other services should not be affected by this issue and we are hoping to restore the control panels to normal operation as quickly as possible. We apologise for any inconvenience.
Update - This issue has now been resolved.
