Gradwell Service News

PROBLEM DESCRIPTION

Telehouse Power Failure

Several rooms in Telehouse North lost power at approximately 14:10 today.
This means that :

Some networks, or portions of networks with particular dependency on
Telehouse North will be off the air.
Your connection to some of your ISPs may terminate on equipment in
Telehouse North which has failed.
All networks that are unaffected will be handling much higher volumes
of traffic,leading to higher latency and packet loss even on connections
that are still available.

Affected Services:

All services

Customer Impact:

Dependant on users ISP peering/routing

Estimated Resolution Time:

We will update again by 16:30

***Update 15:13***

All systems should now be clearing and getting back to normal in Telehouse. Some routing issues will still remain for a little while longer and links will be seeing a much higher throughput as traffic has been re-routed.

***Closing Update 15:51***

Most peering links are now stable and disruption is minimal but please do be aware there still may be slight congestion. Our systems remained online and are not experiencing any problems caused by the earlier power outage.

PROBLEM DESCRIPTION

Affected customers have been experiencing lines that are in Sync but not authenticating for extended periods of time (up to an hour).

Affected Services:

DSL users

Service degredation only, no consistent impact on connectivity.

Customer Impact:

Lines will be connected, however affected customers will not be able to connect to the internet.

Delays in obtaining PPP sessions only.

Estimated Resolution Time:

In progress

Next update at: 14:15

***Update***14:16

Still under investigation. Next update: 15:15

***Update***15:15

Investigations are continuing with 3rd line engineers. Next update: 16:45

***Update***15:56

IP Engineering have performed various tunnel resets from certain Redback LACs across the NUP network. Customers who raised incidents this morning are now in service. B2B are performing further testing with customers this afternoon.

***Update***18:10

Engineers are now performing tests with customers to confirm that this issue is resolved. If you are seeing any further issues please contact support.

***Update*** 09:05 09/07/2010

The outcome will be evaluated this morning.

***Update*** 14:36

3rd Line IP have made progress in resolving customer PPP issues. We will continue to monitor and feed back any related issues.

***Update***

All issues relating to PPP are believed to have been resolved by clearing L2TP tunnels on the isolated redbacks this. 3rd Line IP engineering have opened a case with vendor to identify the root cause. B2B will continue to provide feedback should there be further customers affected by stale sessions.

PROBLEM DESCRIPTION

We are currently seeing issues with routing of customer’s public IP address ranges.

Affected Services:

Gradwell DSL

Customer Impact:

Customers will not be able to access some of their Gradwell DSL public IP address range from external networks.

Estimated Resolution Time:

Our system admins are looking into the issue now and we will post a further update at 11:00am.

This issue has now been resolved for all customers using the Gradwell DSL product. We apologise for any inconvenience caused.

PROBLEM DESCRIPTION

DSL authentication problems

At approximately 08:45 BST, Gradwell’s system administrators were alerted to a problem with the DSL radius server not accepting authentications/logins

Affected Services:

Gradwell DSL

Customer Impact:

If DSL users are currently logged in the session will remain, however anyone trying to connect or reconnect will fail. This will show as the DSL line ’syncing’ but not actually logging on

Estimated Resolution Time:

We are currently investigating the issue and will update again in approximately one hour

This issue has now been resolved. It seems this problem was caused by a small misconfiguration in some new interconnects provided by our DSL partner. This has now been fully resolved and we apologise for any problems this may have caused.

This is advanced notice that we will be performing maintenance to our DSL platform on 10/06/2010 at 02.00 am.

We do not expect this maintenance to last for more than 2 hours.

Thursday, 10th June, 2010

2AM to 4AM BST

Affected Services:

DSL Platform, customers may not be able to establish a connection if their PPP session terminates.

Customer Impact:

The window for this maintenance is scheduled to last 2 hours, during this time NEW connections to our DSL platform may not be possible, existing connections SHOULD not be affected.

Maintenance Action:

This work is being performed to expand the capacity on the DSL platform and also to introduce additional QoS services on our network.

In order to benefit from these enhancements that we are making, you will need to re-establish your DSL session after the maintenance window closes by simply power cycling your router.

Connections that are not re-established will continue to work on the existing service.

We have been informed by our ADSL carrier that customers who connect to the following LLU exchanges will experience a total loss of connectivity.

• Merton
• Bayswater
• Epsom
• Shoreditch
• Malden
• Lower Holloway
• Lea Valley
• Wandsworth
• Wimbledon
• Willesden
• Kingston
• Putney
• Surbiton
• Southbank
• Ponders End

Engineers are currently investigating the cause of this outage, and further updates will be posted once available from our carrier.

** 10:50 - On site engineers have restored service to this exchange. Customers should now be able to re-establish a connection.

We have received the following communication from BT Openreach regarding a National Datastream service outage affecting upto 37,500 circuits.

Customers may experience issues with their ADSL connection, the router will be able to sync and obtain a connection , yet traffic will not pass (Web/Email/VoIP services will be unavailable).

Updates when received from BT Openreach will be posted here.

** 12:17 : 437 exchanges have been confirmed as affected by this outage, Customer Services have received a document detailing affected exchanges.

** 12:30 : BT have taken the decision to close North Paddington Exchange due to flooding , LLU services provided by Tiscali at this exchange only are experiencing the same issues as above.

***Update 13:45***

A full list of the affected exchanges can be found here - http://bit.ly/bedra6

** 17:30 : The first of 2 generators that BT have ordered for North Paddington Exchange has arrived onsite, once flood water has been cleared engineers will be tasked to install it.

***Update 20:25***

We have had an update from BT. They are saying that it may take up to 4 days to resolve the fault. They will be able to give a more definite time frame to fix once they are allowed inside the site by the fire department.

** 08:30 Tiscali LLU Services at North Paddington have been restored. Datastream connectivity issues are still ongoing

** 12:15 Our carrier has announced that connectivity on rdg-0-dsl, he-1-dsl and he-2-dsl has been restored. Connectivity will be monitored for the next 6 hours. Any further updates received will be announced here. Customers should now be able to reconnect to our service, customers may need to power cycle their ADSL router/modem in order to regain connections.

***Update 11:24 6th April***

DSL service should now be back to normal so we are happy to close this status update. If you are seeing any problems with DSL, please raise a support ticket and we will deal on a case-by-case basis.

We are aware of a fire at a BT site in North Paddington, London and this is likely to affect the routing of calls that would have passed through that node. This will affect all communication providers that route through this node.

We have no reports of this affecting any of our users at present but are closely monitoring the situation and have contingency plans in place.

***Update 11:20***

We have lost connection to one exchange but our voice routes have been rerouted where necessary so no ill effects should be seen by any users. We will however leave this status update ‘at risk’ pending updates.

***Update 14:40***

We are starting to see some congestion over some of the affected inbound BT routes. Unfortunately this may last until about 16:00 when call volumes start to drop. We apologise for any problems this may be causing you and please be assured we are doing everything we can to make use of alternative routing to keep this disruption to an absolute minimum.

***Update 15:40***

Congestion issues should be near zero now, we have re-routed the majority of the traffic over alternative routes and call volumes are dropping off. We are still awaiting a further update from BT

***Update 16:00 1/4/2010***

We have had word from BT that the main interconnects have been restored but there will be congestion for some time until BT central core manages to clear. We will continue to monitor

***Update 11:20 6th April***

We are now seeing little or no congestion on BT routing so we are happy to close this status update. If you are still seeing any affected numbers, please raise a support ticket and we will deal with these on a case-by-case basis.

UPDATE 10:23am - This issue has now been resolved and connectivity restored, the root cause was a supervisor card failing to switchover to a resilient card.

We have been informed by our ADSL carrier, that a multiple number of exchanges have lost backhaul connectivity. BT and Opal 3rd line IP engineers are investigating and further updates will be posted.

A list of affected exchanges can be found below

• wandsworth
• merton-park
• richmond-kew
• north-finchley
• ponders-end
• epsom
• putney
• shoreditch
• surbiton
• malden
• willesden
• wimbledon
• aberdare
• worcester-park
• tottenham
• cricklewood
• kingston
• bayswater
• hendon
• teddington
• lower-holloway
• walton-on-thames
• southbank

We have had a notification from our DSL carrier that multiple LLU exchanges are showing as down around the Luton area. Affected customers will not have any services that run over DSL Their TMC are investigating and we will update when further information is made available. Please note, this will only affect LLU lines, IPstream and Datastream lines are unaffected

***Update 16:40***

Our DSL carrier has advised these faults are now totally cleared. We apologise to any of our DSL users that were affected.