Update 21:48: We have brought this server back online and are continuing to migrate all data from it, as this serever remains at risk. Our internal monitoring systems show that lon-web-1 and lon-web-2 have now recovered.

Update: June 30th 23:00: We have now migrated enough from our legacy system ‘Azure’, to mark our new HP/Lefthand copy active and do not expect any more outages with Azure to to affect back-end scripts from running and we therefore consider this issue closed.

Update: This issue has been resolved since 21:30 hours.

Update: Apologies for some additional DNS resolution issues which were encountered after this this point which delayed resolution of the service.  The service was restored by 22:50.

This issue has now been resolved, due to some configuration changes on our core database server cluster.

Update 16:25: Our core storage server has now come back online, however, some of the dependent services, such as webmail, some clustered SMTP outbound and web servers have some file system errors are being restarted.

Individual customer mailboxes or sites are not affected due to our migration to HP/Lefthand storage.

RESOLVED by 16:51: We have now restarted any affected machines, core VoIP services were not affected, although we have had to restart both of our inbound Asterisk-based fax servers.  The majority of customers are not using these.  Our web load balancer was briefly unavailable, which will have caused a brief outage with customer hosted web-sites.  Our monitoring system is continuing to report some machines which have had file system errors and we are intervening as appropriate, however, the majority of these are clustered services, and so customers should not see any effects.

Customer web-sites and control panels were not affected.

Additionally mail will be temporarily unavailable for customers with mailboxes on the mail servers ‘Badlands’, ‘Yosemite’, ‘Denali’ and ‘Yellowstone’. We will be working on these servers individually in order to reduce down time to a minimum.

This maintenance is now in progress.

Update 22:37 - We have now migrated the mail servers and work on the home file stores has started.
Update 23:14 - We have now finished work on customer data stores are are looking at some remaining issues with mail servers.

Update 01:00 -
All services are currently live, due to unexpected networking issues, the mail file servers have not been migrated at this time, but will continue to be active on the old installation.

Update 18:40: The issue above will have caused webmail access to fail from 16:30 until 18:25.

Update - 20:00- We are seeing some issues with the web clusters or rather the data that feeds the web clusters. Some visitors will be greeted with 403 error pages instead of content at present. Our server team are still working hard on this and will implement a fix as soon as possible.

Update - 21:00 This issue has now been resolved.  Our systems admins will be continuing to monitor this issue throughout the evening.  The 403 forbidden errors customers may have seen were due to an incorrect networking configuration on one of our new file storage servers.  The internal issue for this is 1831.  We apologise for any inconvenience caused by this problem, and we will be investigating how best this type of problem can be avoided in the future.