Gradwell Service News

PROBLEM DESCRIPTION

Due to Virgin Media’s power outage yesterday our helpdesk is currently experiencing problems with their telephone lines. We hope to have this resolved by 12:00pm today and be back up to full working order. If any customers experience any issues telephoning Customer Services we would advise you to email support@gradwell.com and we will be able to respond accordingly. We apologise for any inconvenience caused.

Affected Services:

Gradwell Customer Services

Customer Impact:

Intermittent issues contacting Gradwell Customer Services.

Estimated Resolution Time:

12:00pm

**UPDATE**

The telephone lines are now working correctly again. We apologise for any inconvenience caused.

PROBLEM DESCRIPTION

We are experiencing a connectivity issue to our Bath office which has resulted in our support phone lines and email being temporarily unavailable

Affected Services:

Gradwell support

Customer Impact:

Gradwell are currently unable to provide phone or email support

Estimated Resolution Time:

We are working with our Internet providers to restore connectivity.

11:48: Our internet providers are currently working to resolve this fault caused by a power outage at their network facility at London

12:20: Our support team are now able to respond to email only. Please email support@gradwell.com with any support requests while we continue to work on restoring phone support.

14:04: At this time, our telephone support is back online, we are still awaiting our primary connectivity to be restored and will leave this status post open until that time.

17:15 Our Primary connectivity has now been restored, issue resolved.

PROBLEM DESCRIPTION

Telehouse Power Failure

Several rooms in Telehouse North lost power at approximately 14:10 today.
This means that :

Some networks, or portions of networks with particular dependency on
Telehouse North will be off the air.
Your connection to some of your ISPs may terminate on equipment in
Telehouse North which has failed.
All networks that are unaffected will be handling much higher volumes
of traffic,leading to higher latency and packet loss even on connections
that are still available.

Affected Services:

All services

Customer Impact:

Dependant on users ISP peering/routing

Estimated Resolution Time:

We will update again by 16:30

***Update 15:13***

All systems should now be clearing and getting back to normal in Telehouse. Some routing issues will still remain for a little while longer and links will be seeing a much higher throughput as traffic has been re-routed.

***Closing Update 15:51***

Most peering links are now stable and disruption is minimal but please do be aware there still may be slight congestion. Our systems remained online and are not experiencing any problems caused by the earlier power outage.

PROBLEM DESCRIPTION

A backlog of mail is queued on our internal mail delivery servers, causing long delays in delivering mail to customer mailboxes.

Affected Services:

Email.

Customer Impact:

Email which has been sent to your mailboxes may be delayed or appear to be missing.

Estimated Resolution Time:

We are currently investigating this problem and will update by 13:00 with more information. Resolved at 11:42

***Update 09:55***

Two of our four servers have now cleared their backlog of email. We are working to clear the backlog on the remaining two servers. Any new mail now being received should not be experiencing any delay.

***Update 11:10***

The two remaining mail servers are continuing to clear their mail queues. We are currently investigating why this is taking longer than expected.

***Update 11:42***

All mail queues are now fully cleared and mail should be flowing as expected. Some users may be seeing some duplicate messages being delivered.This is caused by inbound mail servers redelivering partially deferred messages resulting in multiple copies. We apologise for any problems caused by these delays.

PROBLEM DESCRIPTION

At approximately 16:35 BST, Gradwell’s system administrators were alerted to a problem with one of our storage SANs

Affected Services:

All services

Customer Impact:

All services may be affected, including VOiP and Hosting. Loss of phone registration and inability to call out or receive calls. Some websites will be offline.

Estimated Resolution Time:

We are investigating this now and will update by 17:45 with more information

***Update 17:00***

All lines into our support office are also offline at present

***Update 17:20***

Services are now starting to return to normal as we are restarting many affected services and servers. Our support line is now back online

***Update 18:15***

Most servers have now been restarted and most services are back on line. We are continuing to resolve any remaining issues.

***Update 18:31***

All services should now be available.

PROBLEM DESCRIPTION

Affected customers have been experiencing lines that are in Sync but not authenticating for extended periods of time (up to an hour).

Affected Services:

DSL users

Service degredation only, no consistent impact on connectivity.

Customer Impact:

Lines will be connected, however affected customers will not be able to connect to the internet.

Delays in obtaining PPP sessions only.

Estimated Resolution Time:

In progress

Next update at: 14:15

***Update***14:16

Still under investigation. Next update: 15:15

***Update***15:15

Investigations are continuing with 3rd line engineers. Next update: 16:45

***Update***15:56

IP Engineering have performed various tunnel resets from certain Redback LACs across the NUP network. Customers who raised incidents this morning are now in service. B2B are performing further testing with customers this afternoon.

***Update***18:10

Engineers are now performing tests with customers to confirm that this issue is resolved. If you are seeing any further issues please contact support.

***Update*** 09:05 09/07/2010

The outcome will be evaluated this morning.

***Update*** 14:36

3rd Line IP have made progress in resolving customer PPP issues. We will continue to monitor and feed back any related issues.

***Update***

All issues relating to PPP are believed to have been resolved by clearing L2TP tunnels on the isolated redbacks this. 3rd Line IP engineering have opened a case with vendor to identify the root cause. B2B will continue to provide feedback should there be further customers affected by stale sessions.

PROBLEM DESCRIPTION

At approximately 6:50  BST, Gradwell’s system administrators were alerted to some system issues affecting a multitude of Gradwell services.

Affected Services:

Web services

Control panels

VoIP services

Customer Impact:

Affected customers will be seeing errors when accessing hosted websites/control panels and will be seeing errors when attempting to make outbound calls.

Estimated Resolution Time:

Our system admin team are working on this now and will update again at or before 09:30.

***Update*** 9:35

VoIP services and control panels should now be working as expected, we are still working on the web clusters and expect to have these back online shortly. We will update again at or before 10:30

***Update*** 10:36

The web cluster is now back online and all services should be running as expected.

There may be some slowdown on control panels as systems are busy processing any backlogs.

We are continuing to monitor and will update again at 11:30

***Update*** 11:29

All systems are running correctly and remain stable. We will continue to monitor closely for the next few hours and update/close this status at 13:30

***Update*** 13:20

All systems are now running correctly and we are now closing this status update.

The problem has been identified as being one of our DNS cache servers. This cache server, 193.111.200.191, stopped responding and this in turn caused our master MySQL server to effectively lock up. This then failed to respond to queries correctly. The majority of our infrastructure relies on this database, hence parts of it became unstable.

We apologise for any problems this has caused you.

Scheduled Maintenance for mail servers; 29/06 @ 10:00pm

This is advance notice that we will be performing maintenance to the mail-server “glacier” on 29/06/2010 at 10:00pm.

We do not expect this maintenance to last for more than 1 hour.

Tuesday, 29th June, 2010

Affected Services:

Customers whose mailboxes are stored on the “glacier” mail server.

Customer Impact:

Customers on the affected mail server who may not be able to access their mailboxes for a short period during the maintenance window. Customers on all other mail servers will not be affected.

Maintenance Action:

This work is being performed to increase the storage space available on the mail server.

PROBLEM DESCRIPTION

We are currently seeing issues with routing of customer’s public IP address ranges.

Affected Services:

Gradwell DSL

Customer Impact:

Customers will not be able to access some of their Gradwell DSL public IP address range from external networks.

Estimated Resolution Time:

Our system admins are looking into the issue now and we will post a further update at 11:00am.

This issue has now been resolved for all customers using the Gradwell DSL product. We apologise for any inconvenience caused.

PROBLEM DESCRIPTION

DSL authentication problems

At approximately 08:45 BST, Gradwell’s system administrators were alerted to a problem with the DSL radius server not accepting authentications/logins

Affected Services:

Gradwell DSL

Customer Impact:

If DSL users are currently logged in the session will remain, however anyone trying to connect or reconnect will fail. This will show as the DSL line ’syncing’ but not actually logging on

Estimated Resolution Time:

We are currently investigating the issue and will update again in approximately one hour

This issue has now been resolved. It seems this problem was caused by a small misconfiguration in some new interconnects provided by our DSL partner. This has now been fully resolved and we apologise for any problems this may have caused.