Since 10:50 One of our legacy ESX servers has experienced an issue communicating with our legacy storage servers which has stopped several servers operating correctly. The only remaining server experiencing an issue is v-mail-file-10, which holds 12 mailboxes which are the only mailboxes not yet migrated on to our HP/Lefthand storage. This server has been brought back up on an alternative host and is currently undergoing a file-system test before being brought back online. We apologise for any inconvenience caused.
Update: This issue was fully resolved at 12:30
18:50 We are currently experiencing an issue with one of the servers which provides our NNTP/usenet feeds and our server team is investigating.
Update: This issue has been resolved since 21:30 hours.
Update: Apologies for some additional DNS resolution issues which were encountered after this this point which delayed resolution of the service. The service was restored by 22:50.
We are currently investigating an inbound mail delay to hosted domains and email forwarding. This issue has been narrowed down to a firewall configuration issue and is being worked on by our on-call server engineers. We apologise for any inconvenience caused.
Update 23:30: The root cause of this issue has now been corrected and mail queues are being cleared down, which is likely to last around half an hour.
Update 00:47: An additional issue has been found with delivery to local mailboxes from the affected inbound servers and has now been corrected.
Update: This issue was resolved by 02:00.
We have identified a DNS issue which affects access to our core database services. This issue will affect call set up times and access to our control panels. Our server team are currently investigating this problem and will update gradwellstatus as soon as possible.
This issue has now been resolved, due to some configuration changes on our core database server cluster.
We have identified an issue with one of the servers in our outbound mail cluster, which was no longer handling outbound mail due to a configuration error. This has now been corrected, customers whose mail has been routed via this server may experience some delays with outbound email. We apologise for any inconvenience caused. This issue affects around 20% of sent email.
We are currently experiencing an outage of one of our core storage servers - Europa which will temporarily affect access to our part of our web-services and some other services. We will publish more information as soon as it is available.
Update 16:25: Our core storage server has now come back online, however, some of the dependent services, such as webmail, some clustered SMTP outbound and web servers have some file system errors are being restarted.
Individual customer mailboxes or sites are not affected due to our migration to HP/Lefthand storage.
RESOLVED by 16:51: We have now restarted any affected machines, core VoIP services were not affected, although we have had to restart both of our inbound Asterisk-based fax servers. The majority of customers are not using these. Our web load balancer was briefly unavailable, which will have caused a brief outage with customer hosted web-sites. Our monitoring system is continuing to report some machines which have had file system errors and we are intervening as appropriate, however, the majority of these are clustered services, and so customers should not see any effects.
We will be perfoming essential maintenance on our virtualisation platform this evening at 22:00 which will involve home directories temporarily being unavailable for customers on Clearwater and Dixie. This will also mean that web-sites hosted on these servers are temporarily unavailable.
Additionally mail will be temporarily unavailable for customers with mailboxes on the mail servers ‘Badlands’, ‘Yosemite’, ‘Denali’ and ‘Yellowstone’. We will be working on these servers individually in order to reduce down time to a minimum.
This maintenance is now in progress.
Update 22:37 - We have now migrated the mail servers and work on the home file stores has started.
Update 23:14 - We have now finished work on customer data stores are are looking at some remaining issues with mail servers.
Update 01:00 -
All services are currently live, due to unexpected networking issues, the mail file servers have not been migrated at this time, but will continue to be active on the old installation.
We are currently restarting our main web-site server due to an issue customers have been experiencing with access to webmail. We apologise for any inconvenience caused by this essential emergency maintenance. This will not affect customer hosted web-sites.
Update 18:40: The issue above will have caused webmail access to fail from 16:30 until 18:25.
Our engineers are currently investigating a fault affecting some pop3 and imap mailboxes that is preventing customers logging in to read their email. We will update here as soon as we have further information.
UPDATE: All issues affecting pop3 and imap have now been resolved, we are not aware of any outstanding issues.
We are currently restarting the customer mail server; v-mail-file-7
Mail sending for customers using IMAP and POP3 and IMAP collection will be temporarily interrupted, we apologise for any inconvenience caused.
Please note: On certain email clients, you may need to re-enter your password once the server is running again.
Update 09:30: This issue has now been resolved. We are continuing to migrate customer data from this platform and on to our HP/Lefthand storage as quickly as possible, however our existing, older storage can be sensitive to moving customer data.
