(39) (9) (65) (63) (63) (10) (12) (3) (110)Starts: 2008/11/25 22:00 Ends 2008/11/26 01:00
We are performing maintenance tonight on our iSCSI-3 storage unit, to increase the size of the internal queues on this storage unit. During this maintenance window, some customers may be unable to access their email via POP3 or IMAP. All incoming email will be queued, and successfully delivered after the maintenance has completed.
We apologise for any inconvenience this maintenance causes.
Our server team are currently investigating a problem with the NewSIP system. Our NewSIP database is not currently available, which will prevent registrations on the system. This service should be restored very shortly and we apologise for any inconvenience caused.
Update 15:30 The VoIP service on newsip.gradwell.net has been restored. Our server team have rebuilt the back-end database and the service has now been working correctly for approximately 30 minutes.
We are still currently experiencing issues with our mail file server v-mail-file-7. This server has hung and will not currently restart. Our server team are working to resolve this problem as soon as possible to restore service and apologise for the inconvenience.
Update 14:30 The mail file server v-mail-file-7 is now up and receiving POP3and IMAP logins correctly. Load on this server is likely to be high due to mail deliveries and customer demand for a little while.
We are currently receiving some reports of inbound numbers from our one of our inbound telcos being unavailable. Our PSTN switch engineers are investigating this issue and we will update customer as soon as possible. We are aware at this stage that the majority of ported numbers are not affected.
Update 16:47: This issue has been corrected by our inbound telco and we are investigating with them what the cause of the problem was. We apologise for any inconvenience caused.
v-mail-file-7 is currently unavailable and customers will not presently be able to access mail on this file server. Our engineers are investigating this and will be restoring the service as quickly as possible. Further updates will be available on www.gradwellstatus.com. We apologise for any inconvenience caused.
Update 10:50 This server is now back in service pending some routing checking our server team are doing to ensure mailboxes are available.
Update 16:40 Unfortunately our mail server v-mail-file-7 is still experiencing some issues and we are currently liasing with our software vendor for an urgent resolution to this problem. We apologise again for the disruption this mail file server is causing to our customers and our technical team will try to help as best we can.
Update 17:42 Our server team are currently building a new file server on our newest SAN to replace the existing one and will be migrating the existing customer data across.
Update 20:00 v-mail-file-7 is now available for customer use. We apologise for all customers affected by this outage for the extended time taken to resolve this problem. Our server team will be investigating the issue with VMware to try to prevent this issue happening in future.
We are currently investigating a problem with customers who have changed their password within the past few months and who have mail on v-mail-file-5. These logins are not currently available via POP3 and this has been raised internally as an urgent issue. IMAP and webmail logins to these mailboxes are not affected. Web mail is available at https://www.gradwell.net/webmail
Update 14:45: This problem has now been resolved and POP3 connections are working correctly. We apologise for any inconvenience to our customers who regularly change their passwords.
We are currently experiencing an issue with v-mail-file-5, one of our customer mail servers. This server is currently offline and is being copied to a new machine and will not be accessible under the issue has been cleared. Further information will be made available as soon as it is received from our server team. We apologise for any inconvenience caused.
Update 16:15 Our server team are currently cloning a new machine to provide access to the migrated data and this should be back online shortly. We are aiming to make this available as quickly as possible.
Update 16:30 Our server team have restored service to v-mail-file-5. We apologise for the interruption caused by this unexpected failure. We have tested IMAP and POP3 logins to a random selection of mail boxes to ensure everything is running correctly.
Some customers are reporting that they cannot reach our sites or services at all. Our investigations have shown that affected customers are using Orange Broadband. We are investigating this issue with our transit providers. We would recommend that customers report this issue to Orange directly. We will update this status announcement as soon as any more information is available.
Update 15:00 We have been in touch with both of our transit providers and they have reported that the issue appears to be with Open Transit/France Telecom (of which Orange is a reseller). Our transit providers have attempted to contact IP addresses in these ranges and are seeing varying degrees of packet loss. If you are experiencing this problem, please contact your broadband provider to report a fault.
Update: 19:30, 23rd Nov: We understand from customers that Orange/FT have now resolved this problem.
We are currently experiencing an issue with our NewSIP platform, which handles call forwarding and our new VoIP platform. Our engineers are working on this issue to resolve the problem as quickly as possible. We apologise for any inconvenience.
Update: Our engineers have worked quickly to resolve this issue and the service has now been restored. The service would not have been available for no more than a few minutes. If customers are having issues with phones or equipment, we would recommend restarting them, however, most customer phones should re-register automatically.
We are experiencing an issue with one of our inbound number providers on what appears to be mainly London numbers. Our suppliers are working with BT to resolve the issue as soon as possible.
16:18 We have received an update from BT that they are investigating one of their exchanges. They have not as yet confirmed which exchange is affected. We apologise for the ongoing inconvenience. Our inbound number provider is pressing BT for updates.
17:17 Our inbound carrier has informed us that BT have now resolved an issue with their exchange and that all affected numbers should be working correctly. We have tested some of the numbers reports and they are correctly calling into our system from BT. If you are still experiencing an issue, please let us know by sumitting an incident through the customer portal.
