Some users on ‘glacier’ are seeing issues collecting mail via POP3/IMAP this morning. Our system admin team are working on this now and we will update here as soon as further information is available.
We apologise for any problems this may be causing you.
***Update 13:55***
We have implemented a fix for the earlier mail problems and have rebalanced all mail stores. We also have further capacity coming online within the next two weeks to ensure these problems do not reoccur.
We apologise again for any problems caused by these earlier issues
We are currently experiencing a problem with one of our VMware hosts. To return full functionality we need to reboot one of our blades. This will cause an interruption to several services including shell.gradwell.net, some websites hosted on ‘dixie’ and hosted exchange services.
We will post here again upon completion of this emergency work. We apologise in advance for any problems caused by this.
***Update 13:54***
We have fully powered off the problem blade and are restarting services now. Some virtual machines will require a filesystem check so these will take slightly longer to complete.
***Update 14:12***
We have now restored all services. If you are still seeing any problems, please do raise an incident with our support team and we will investigate on a case by case basis. Our apologies again for the loss of service.
We have discovered that Microsoft Hotmail/Yahoo/Live networks have decided to block our mailservers from sending outbound mail directly to their MX servers. We have requested removal from this blocklist and are awaiting Microsoft to action.
We apologise for any inconvenience this causes and would like to remind anyone using us to forward mail to one of these accounts to not ‘report as spam’ if the mail is only forwarded through us. This blacklists our network, not the originating network.
***Update 10:10***
We are now only subject to the normal Microsoft ‘throttling’ for all of their inbound servers so the vast majority of emails should now be reaching their intended target. We apologise for any problems resulting from this and we have asked for our sending limits to be raised across all Microsoft servers.
We are currently seeing a massive influx of mail through our inbound mail servers and this is causing delays on mail delivery across the network. Our system admin team are working on this now and we will update here with further details as soon as possible. We apologise for any problems caused by this.
UPDATE: All mail queues on our systems have now cleared and mail delivery is proceeding as normal. Our investigations have highlighted a number of mail sources delivering abnormally large volumes of email whom we have taken action against. Further to this we have highlighted some architectural improvements within our own systems in order to mitigate these types of attacks in the future.
***Update 13:20 12/2***
We are once more seeing a massive amount of inbound mail causing some queues on our mail platform. Our system administration team are working on this now and hope to have any backlog cleared as soon as possible. Again we apologise for any problems caused by this and will update here again once the queues have been fully cleared
***Update 15:58***
Our queues have held at a low level for quite a while now but we are still seeing a huge amount of inbound connections to our edge nodes. Our system admin team are bringing extra resources online to help keep delays to a minimum. We are continuing to monitor the situation and will update here again.
***Update 13/02/10 13:00***
Systems are currently processing new mail well, however, we have identified two servers which have messages stuck in their local delivery queues from Friday. We are manually flushing those messages at the moment so customers may see some delayed messages arriving in their inbox.
***Update 15:40 16/2/10***
We have been closely monitoring the mail platform and the previous delays have been fully cleared and no further delays have been seen.
This was partially caused by header corruption breaking our mail loop detection coding. This has now been recoded and fully tested. We apologise again for any problems caused by these delays.
Our network engineers are investigating an outage with our connectivity to Sovereign House, which may prevent customers from reaching our network and services, although we have not yet received indepdendent confirmation of this. We will keep this page updated as soon as further information is available.
Update 05:51 : This problem was first detected by our monitoring systems just after 4AM. We have confirmed that this is likely to cause VoIP registration issues for some customers. Our network engineers are working with our transit providers to restore connectivity. We are not able to provide an ETA for restoration of our link with Sovereign House at the moment. We apologise for any inconvenience this will be causing customers are are are working to ensure the link is restored by UK business hours.
Update: 06:14: We have opened an internal issue - 3108 for linking customer incidents where problems are proven to be related to this fault. Typical problems involve not being able to send or receive email, and phones not registering. You are still able to reach our web-site at www.gradwell.com
Update 06:50 - This issue appears to be resolved, however, we are awaiting an update from our network team to confirm the status of the resolution.
Update: This issue was resolved just before 07:00 and was related to one of our transit providers’ routers due to overnight maintenance. If you are experiencing any issues with phones this morning, please try to reboot them, many phones will not continually retry if there is a network issue. If problems persist, please contact our support team, who will be happy to advise.
We are seeing some mail delays at present. This was caused by a backlog caused by spam and our system admin team are clearing mail queues now.
We apologise for any problems this is causing you.
***UPDATE 11:05***
We have cleared all spam mail from our mail cluster and our mail queues are now returning to normal
We are currently experiencing an issue where email delivery to our mailing list servers is failing.
This will affect customers who run mailing lists under the subdomain “list.yourdomainname.dom”
An oncall system engineer has been alerted to this issue by our internal monitoring and further updates will be posted once available.
Update: this issue was resolved at 11:45 this morning and was caused by a low-level MTA port conflict on our Mailman server. We cannot be sure why this happened, however, we have changed the configuration to ensure only Exim runs in future. We apologise for any inconvenience caused. Mailman is now processing the backlogged queue and weill attempt re-delivery for all unprocessed messages.
Mailboxes stored on our ‘Yellowstone’ mail store have temporarily been unavailable due to a memory problem. The server has been restarted and is now operational. Customers may be prompted by their email clients for a password when sending or receiving. Our engineers will be performing further maintenance to increase memory of this server this evening at 22:00. We do not expect this maintenance to last more than a few minutes, during which mailboxes will be temporarily unavailable.
Update: This maintenance was successfully completed between 22:25 and 22:35.
We are currently investigating an issue related to outbound email and will update customers as soon as further information is available.
This problem has now been resolved and was identified as a temporary issue with our auditing database. We apologise for any inconvenience.
We are removing SMTP after POP3 support from the network. This was a legacy authentication method for email clients which did not support usernames and passwords. Customers will have received periodic notifications from us that this service would be withdrawn if they were using it. The majority of mailboxes will have this featured removed today, customers who have actively been using it with the past two weeks will have received a separate email today and these customers will be gradually switched over during the next few days. No mailboxes will support this method of authentication by November 1st.
If you have difficulty sending email, please ensure you are using your correct username and password for the outbound SMTP settings as well as the inbound POP3 or IMAP settings.
Update 16:00 Thursday Oct 22nd: The majority of mailboxes have this feature withdrawn, the remaining customers who use this feature will be disabled periodically up to November 1st.
