Gradwell Service News

PROBLEM DESCRIPTION

Telehouse Power Failure

Several rooms in Telehouse North lost power at approximately 14:10 today.
This means that :

Some networks, or portions of networks with particular dependency on
Telehouse North will be off the air.
Your connection to some of your ISPs may terminate on equipment in
Telehouse North which has failed.
All networks that are unaffected will be handling much higher volumes
of traffic,leading to higher latency and packet loss even on connections
that are still available.

Affected Services:

All services

Customer Impact:

Dependant on users ISP peering/routing

Estimated Resolution Time:

We will update again by 16:30

***Update 15:13***

All systems should now be clearing and getting back to normal in Telehouse. Some routing issues will still remain for a little while longer and links will be seeing a much higher throughput as traffic has been re-routed.

***Closing Update 15:51***

Most peering links are now stable and disruption is minimal but please do be aware there still may be slight congestion. Our systems remained online and are not experiencing any problems caused by the earlier power outage.

PROBLEM DESCRIPTION

Affected customers have been experiencing lines that are in Sync but not authenticating for extended periods of time (up to an hour).

Affected Services:

DSL users

Service degredation only, no consistent impact on connectivity.

Customer Impact:

Lines will be connected, however affected customers will not be able to connect to the internet.

Delays in obtaining PPP sessions only.

Estimated Resolution Time:

In progress

Next update at: 14:15

***Update***14:16

Still under investigation. Next update: 15:15

***Update***15:15

Investigations are continuing with 3rd line engineers. Next update: 16:45

***Update***15:56

IP Engineering have performed various tunnel resets from certain Redback LACs across the NUP network. Customers who raised incidents this morning are now in service. B2B are performing further testing with customers this afternoon.

***Update***18:10

Engineers are now performing tests with customers to confirm that this issue is resolved. If you are seeing any further issues please contact support.

***Update*** 09:05 09/07/2010

The outcome will be evaluated this morning.

***Update*** 14:36

3rd Line IP have made progress in resolving customer PPP issues. We will continue to monitor and feed back any related issues.

***Update***

All issues relating to PPP are believed to have been resolved by clearing L2TP tunnels on the isolated redbacks this. 3rd Line IP engineering have opened a case with vendor to identify the root cause. B2B will continue to provide feedback should there be further customers affected by stale sessions.

PROBLEM DESCRIPTION

DSL authentication problems

At approximately 08:45 BST, Gradwell’s system administrators were alerted to a problem with the DSL radius server not accepting authentications/logins

Affected Services:

Gradwell DSL

Customer Impact:

If DSL users are currently logged in the session will remain, however anyone trying to connect or reconnect will fail. This will show as the DSL line ’syncing’ but not actually logging on

Estimated Resolution Time:

We are currently investigating the issue and will update again in approximately one hour

This issue has now been resolved. It seems this problem was caused by a small misconfiguration in some new interconnects provided by our DSL partner. This has now been fully resolved and we apologise for any problems this may have caused.

We have been informed by our ADSL carrier that customers who connect to the following LLU exchanges will experience a total loss of connectivity.

• Merton
• Bayswater
• Epsom
• Shoreditch
• Malden
• Lower Holloway
• Lea Valley
• Wandsworth
• Wimbledon
• Willesden
• Kingston
• Putney
• Surbiton
• Southbank
• Ponders End

Engineers are currently investigating the cause of this outage, and further updates will be posted once available from our carrier.

** 10:50 - On site engineers have restored service to this exchange. Customers should now be able to re-establish a connection.

We have received the following communication from BT Openreach regarding a National Datastream service outage affecting upto 37,500 circuits.

Customers may experience issues with their ADSL connection, the router will be able to sync and obtain a connection , yet traffic will not pass (Web/Email/VoIP services will be unavailable).

Updates when received from BT Openreach will be posted here.

** 12:17 : 437 exchanges have been confirmed as affected by this outage, Customer Services have received a document detailing affected exchanges.

** 12:30 : BT have taken the decision to close North Paddington Exchange due to flooding , LLU services provided by Tiscali at this exchange only are experiencing the same issues as above.

***Update 13:45***

A full list of the affected exchanges can be found here - http://bit.ly/bedra6

** 17:30 : The first of 2 generators that BT have ordered for North Paddington Exchange has arrived onsite, once flood water has been cleared engineers will be tasked to install it.

***Update 20:25***

We have had an update from BT. They are saying that it may take up to 4 days to resolve the fault. They will be able to give a more definite time frame to fix once they are allowed inside the site by the fire department.

** 08:30 Tiscali LLU Services at North Paddington have been restored. Datastream connectivity issues are still ongoing

** 12:15 Our carrier has announced that connectivity on rdg-0-dsl, he-1-dsl and he-2-dsl has been restored. Connectivity will be monitored for the next 6 hours. Any further updates received will be announced here. Customers should now be able to reconnect to our service, customers may need to power cycle their ADSL router/modem in order to regain connections.

***Update 11:24 6th April***

DSL service should now be back to normal so we are happy to close this status update. If you are seeing any problems with DSL, please raise a support ticket and we will deal on a case-by-case basis.

We are aware of a fire at a BT site in North Paddington, London and this is likely to affect the routing of calls that would have passed through that node. This will affect all communication providers that route through this node.

We have no reports of this affecting any of our users at present but are closely monitoring the situation and have contingency plans in place.

***Update 11:20***

We have lost connection to one exchange but our voice routes have been rerouted where necessary so no ill effects should be seen by any users. We will however leave this status update ‘at risk’ pending updates.

***Update 14:40***

We are starting to see some congestion over some of the affected inbound BT routes. Unfortunately this may last until about 16:00 when call volumes start to drop. We apologise for any problems this may be causing you and please be assured we are doing everything we can to make use of alternative routing to keep this disruption to an absolute minimum.

***Update 15:40***

Congestion issues should be near zero now, we have re-routed the majority of the traffic over alternative routes and call volumes are dropping off. We are still awaiting a further update from BT

***Update 16:00 1/4/2010***

We have had word from BT that the main interconnects have been restored but there will be congestion for some time until BT central core manages to clear. We will continue to monitor

***Update 11:20 6th April***

We are now seeing little or no congestion on BT routing so we are happy to close this status update. If you are still seeing any affected numbers, please raise a support ticket and we will deal with these on a case-by-case basis.

UPDATE 10:23am - This issue has now been resolved and connectivity restored, the root cause was a supervisor card failing to switchover to a resilient card.

We have been informed by our ADSL carrier, that a multiple number of exchanges have lost backhaul connectivity. BT and Opal 3rd line IP engineers are investigating and further updates will be posted.

A list of affected exchanges can be found below

• wandsworth
• merton-park
• richmond-kew
• north-finchley
• ponders-end
• epsom
• putney
• shoreditch
• surbiton
• malden
• willesden
• wimbledon
• aberdare
• worcester-park
• tottenham
• cricklewood
• kingston
• bayswater
• hendon
• teddington
• lower-holloway
• walton-on-thames
• southbank

We have had a notification from our DSL carrier that multiple LLU exchanges are showing as down around the Luton area. Affected customers will not have any services that run over DSL Their TMC are investigating and we will update when further information is made available. Please note, this will only affect LLU lines, IPstream and Datastream lines are unaffected

***Update 16:40***

Our DSL carrier has advised these faults are now totally cleared. We apologise to any of our DSL users that were affected.

Several LLU exchanges connected to the Shareshill aggregation router have lost connectivity due to a backhaul fault, customers connected to these exchanges will not have any DSL connection available.

All exchanges will still have 999 emergency call functionality.

BT are currently investigating. Exchanges affected have been listed below.

Broadwell, Great Barr, Stourbridge, Kingswinford, Kidderminster, Tipton and Dudley.

UPDATE 13:34 - We have been informed that BT engineers are onsite carrying out fibre repairs , further updates will follow.

UPDATE 15:09 - BT Engineers are continuing to repair the fibre that has become damaged.

UPDATE 09:12 25/2/10 - Services have now been restored to all exchanges above apart from Tipton, BT engineering are continuing to investigate and repair fibre.

UPDATE 09:02 26/2/10 - Services to all exchanges listed above have now been restored.

Some users might be seeing some problems making a connection to our Gradwell DSL service. This is being investigated now by our on-call system admin team and our core network engineers. We will post any updates here as soon as possible.

We apologise for any problems this is causing

Update 13:53 - Our network engineers have discovered a problem on our DSL termination equipment which did not appear to have responded to a fault earlier in the day which meant that one of our RADIUS servers was no longer available.  The fault has been corrected and we will be working with our hardware vendors to determine what the best course of action is to ensure this type of fault cannot happen in future.