RESOLVED: Customer MySQL DB server and PBX updates
PROBLEM DESCRIPTION
voip-manager and mysqldb are currently experiencing an outage.
At approximately 18:00 BST, our systems team became aware of an issue affecting both our provisioning and one of our customer-facing database servers. This fault appears to have started at around 17:00, and was a progressive problem, as the relevant servers became less responsive over time. We are presently investigating the fault, and working with our Telehouse operations team to rectify the problem as soon as possible.
Affected Services:
PBX provisioning
Customer databases on mysqldb.gradwell.com
Customer Impact:
Whilst all VoIP services are up, changes cannot be processed, so any updates performed via the control panels will not become live.
All customer databases hosted on the mysqldb.gradwell.com service. This will not affect other servers, such as mysql5db-1 or our other two MySQL 4-based servers.
Estimated Resolution Time:
We are presently awaiting on-site engineers restarting the affected services and will provide an update within an hour.
***Update 19:38***
Our on-site ops team have been unsuccessful in attempts to restart the machine so this would appear to be a hardware failure. Our sysadmin and VoIP dev team are currently building a replacement for this machine. At present we are unable to offer a concrete ETA but we will update again as soon as possible with any updates. Our next scheduled update will be in approximately one hour.
***Update 21:26***
VoIP: Our team have now restored our VoIP provisioning systems. If any further issues are seen with VoIP-related updates, please contact our support team.
Hosting: Due to this hardware failure, we are restoring database access to a recent backup (approximately 1AM this morning). Changes to databases on mysqldb.gradwell.com today will have been lost. Other database servers are not affected, and we apologise for any inconvenience this may cause you. This work should complete shortly.
*** Update ***
This issue was resolved on the day the alert was announced on www.gradwellstatus.com It has now been closed as a historical problem. Please note that the replacement MySQL server runs MySQL 5










