(39) (9) (65) (63) (63) (10) (12) (3) (108)One of our customer facing caching DNS resolvers, with address 193.111.200.91 was not responding to DNS queries from around 04:00 until just after 07:00 today. Customers are advised to configure computers and phones with at least two resolver IP addresses:
193.111.200.91 and 193.111.200.191
We have corrected the problem and reported this fault back to the engineers who set up these machines. We apologise for any inconvenience caused.
We are in the process of rebooting one of our ESX machines ‘Mars’, this will affect several services including our public DNS resolvers.
VoIP Customers may experience DNS related issues if phones are configured to use 193.111.200.91 and 193.111.200.191 as their DNS Servers. DSL Customers may experience an outage where web sites cannot be reached.
Our server admin team are working on this now and will try to keep disruption to a minimum. Our apologies for any problems experienced because of this
Update 12:51: Our DNS resolvers are now back online and working correctly. This issue will have resulted in a problem setting up calls. We are currently bringing back other services such as control panels are working as quickly as possible to restore any affected services.
Update 13:16: Control panel access has been restored.
Some users are seeing duplicates of the same zone file within the hosting control panel.
If you are affected by this, we would ask you to not make any changes until we have resolved this as any change made may possibly corrupt the zone file.We will post an update here as soon as possible.
We apologise for any problems this causes you.
****Update 10:26****
This issue is now totally cleared. If you are seeing any oddities or missing any zone files, please raise an incident with support and we will investigate and restore.
Our primary nameserver ns0.gradwell.com is currently not responding to queries.
Our server admin team are working on this now and we will post an update here as soon as possible. Our secondary nameservers are functioning without issue.
We apologise for any problems this is causing
***Update - 15:38***
ns0 is now back up and running and resolving correctly.
Starts: 2009/03/15 22:00 Ends: 2009/03/15 23:59
On Sunday night we will be moving our authoritative DNS generation and synchronization onto a new dedicated server. As part of this work we will be reconfiguring the DNS software on all of our authoritative DNS servers, and as such each server may suffer brief service interruptions, however we will keep such downtime to a minimum and ensure that only one server is affected at any given time, as such there should be very little impact to customers.
This work will affect all hosted domains, using primary and secondary DNS, on our own nameservers (ns0/1/2.gradwell.net) as well as any domains customers are hosting on dedicated nameserver addresses. This work will also affect the gradwell.net domain and so there is potential for minor disruption across all services.
Additionally, during the maintenance, customers may find that DNS updates from the control panel and our DynDNS service may not take effect immediately.
Please accept our apologies for any inconvenience this work may cause, however it is essential maintenance to improve the resilience of our DNS service.
Update: This work was completed at approximately 23:30 on Sunday night with minimal service disruption.
Starts: 2009/02/11 18:00 Ends: 2009/02/11 23:59
Newred will be unavailable for two separate periods on Wednesday evening whilst we replace a faulty hard drive and rebuild the RAID array. During this time, changes to DNS will be delayed, dynamic DNS may be unreliable, email collection from remote servers will be unavailable, and customers won’t be able to log into this server via SSH and FTP.
We apologise for the short notice for this maintenance.
We are currently investigating a DNS issue across our network which will be causing a wide range of not found errors for all services. We are working as quickly as possible to resolve this and will update customers as soon as possible.
Update 18:30 - This issue has now been resolved and an internal issue raised to ensure this does not occur again. We have identified a single point of failure in our zone configuration and our server team will be working to provision backup.
Update 18:55 - One of our inbound telcos for ported numbers have cached a copy of our resource records which is incomplete. This is causing inbound calls from this provider towards our network to fail. We are awaiting an emergency call out from their out of hours engineers to resolve this and will update customers as soon as we have any further news.
Update: 19:30 - A technican at our inbound telco for ported numbers has restarted their DNS cluster to removed the affected DNS zone. This has resolved the issue for inbound numbers failing.
We apologise for the extended nature of this problem, which is due in part to DNS caches not all adhering to the maximum lifespan we would like to give them. This is a problem built into the internet by design, however, we should not have released the non-complete zone which triggered this to happen. As a result, we will be changing the way our DNS servers are configured to try to prevent this issue from happening in the future.
We are currently experiencing an issue with our back-end database server which we are working to resolve very shortly. We apologise for any inconvenience caused. This may result in call setups failing, however, active calls will not be dropped.
Update 12:35
We have restored database connectivity. This issue was caused by a DNS issue internally which is currently being investigated in more details to ensure that more redundancy is added to ensure this does not happen again.
We are still seeing issues with whitelabel VoIP partners’ control panels which we are investigating currently.
Update 13:45
Our partner login areas are now working normally again, this issue is related to our database, which due to being unavailable had a backlog of work, which has now cleared. We apologise for the inconvenience this will have caused you.
Starts: 2008/08/12 22:00 Ends: 2008/08/12 23:59
Late on Tuesday evening, we will be carrying out some essential maintenance to our hosting service.
