***Update 16:40***

Our DSL carrier has advised these faults are now totally cleared. We apologise to any of our DSL users that were affected.

All exchanges will still have 999 emergency call functionality.

BT are currently investigating. Exchanges affected have been listed below.

Broadwell, Great Barr, Stourbridge, Kingswinford, Kidderminster, Tipton and Dudley.

UPDATE 13:34 - We have been informed that BT engineers are onsite carrying out fibre repairs , further updates will follow.

UPDATE 15:09 - BT Engineers are continuing to repair the fibre that has become damaged.

UPDATE 09:12 25/2/10 - Services have now been restored to all exchanges above apart from Tipton, BT engineering are continuing to investigate and repair fibre.

UPDATE 09:02 26/2/10 - Services to all exchanges listed above have now been restored.

We apologise for any problems this is causing

Update 13:53 - Our network engineers have discovered a problem on our DSL termination equipment which did not appear to have responded to a fault earlier in the day which meant that one of our RADIUS servers was no longer available.  The fault has been corrected and we will be working with our hardware vendors to determine what the best course of action is to ensure this type of fault cannot happen in future.

VoIP Customers may experience DNS related issues if phones are configured to use 193.111.200.91 and 193.111.200.191 as their DNS Servers. DSL Customers may experience an outage where web sites cannot be reached.

Our server admin team are working on this now and will try to keep disruption to a minimum. Our apologies for any problems experienced because of this

Update 12:51: Our DNS resolvers are now back online and working correctly.  This issue will have resulted in a problem setting up calls.  We are currently bringing back other services such as control panels are working as quickly as possible to restore any affected services.

Update 13:16: Control panel access has been restored.

Geographic location of affected services:

porthcawl

thatcham

newbury

alton

newbury

basingstoke

thatcham

alton

tadley

tadley

caversham

caversham

porthcawl

basingstoke

Engineers and power contractors are onsite and have identified an issue which they are now working to resolve.

Further updates will be posted once received from our carrier

UPDATE 12:16 - All sites affected are still without connectivity, our carrier and its contractors are expecting the next update in an hours time.

UPDATE 13:42 - Connectivity has been restored to all affected sites, however this may be inturrupted as engineering works are ongoing to correct the original fault.

UPDATE 16:59 - This issue has now been resolved, connectivity to all affected sites has been fully restored.

]]> http://www.gradwellstatus.com/2009/09/30/in-progress-dsl-exchange-outage/feed/ http://www.gradwellstatus.com/2009/09/24/in-progress-core-network-issues/ http://www.gradwellstatus.com/2009/09/24/in-progress-core-network-issues/#comments Thu, 24 Sep 2009 07:57:50 +0000 Ben Smithurst http://www.gradwellstatus.com/?p=1726 ADDED: 08:30 24th Sept

We have been experiencing intermittent periods of poor network performance over the last 24 hours.

This is being caused by a fault that has developed on our main cisco switch stack in which it’s CPU is running at 100%, typically for intermittent bursts of 4-5 minutes.

We have esclated this issue to the Cisco Assistance Centre and we are anticipating their diagnosis this morning. We will then be able to move forward.

If hardware replacement is necessary, we can complete this within 4 hours, likely at the end of Thursday’s business day.

Please accept our apologies for this fault and be assured that it is being pursued.

UPDATED: 09:30 24th Sept

Our network managment suppliers have escalated our Cisco case to “P1″ status, and are currently waiting on their response.

We have also lowered the CPU load on our core switches which has significantly reduced the packet loss on the network, and we are currently taking steps to isolate our voice network onto seperate routing equipment.

UPDATED: 10:45 24th Sept

Our network is fairly stable, but we are still seeing some issues with call quality and broadband speed.

Cisco are currently working on a resource exhaustion theory, where by the reload of our switch last wednesday has triggered a bug that has manifest itself slowly over the week.  We are currently probing their diagnosis, but there is a possibility that a further switch outage of approximately 10 minutes later today may be necessary in order to correct the problem.

UPDATED: 13:00 24th Sept

Following expert advice from Cisco and our network management consultants we have identified that the problem is due to memory allocation (TCAM profile) on the Cisco switches we use.

This meant that if our network was being scanned (for example by a virus or potential intruder) then part of the processor ran out of memory and caused CPU load to become very high, causing problems with the switches performance.

We have disabled a number of features on these which has made more memory available and will monitor the switch performance this afternoon, as it currently seems stable.

It may be necessary to reboot the switch, but we can now perform this out of hours.

We will update this status message again at 18:00 unless a further problem develops.

Update 18:00: We believe that this issue is now fully resolved after configuration changes were made around 13:00 this afternoon, and we are happy that the core network is now stable.

Update 20:20: We are currently still experimencing problems with our core network and are awaiting urgent information from our network engineers.

Update 20:40: Our network team are restoring our switch configuration to the known working configuration prior to last Wednesday’s scheduled maintenance and will be performing an emergency reboot of the switch shortly, which will result in a brief outage.

Update: 21:25: We have seen a decrease in latency over our core network since 20:45, however, some packet loss is continuing.  Our network team will restart our switches with a reverted configuration within the next few minutes.

Update 21:45: Our network team have now restarted our core switches with an older configuration.  All services should be working normally and our internal monitoring is reporting success for all services.

Please note, this is not affecting the ability for customers to connect to our DSL platform.

Further updates will be posted as they are received from our carrier.

This issue has now been resolved by our carrier and all systems are now working correctly.

UPDATE 14:25 IP Engineers have rectified this issue, connectivity has been restored to all affected exchanges and confirmed stable.

The list of affected exchanges have been listed below for reference

• barnstaple
• bideford
• bodmin
• brixham
• crediton
• dawlish
• exeter
• exmouth
• honiton
• ilfracombe
• kingsbridge
• newton-abbot
• newtonabbot
• ottery-st-mary
• otterystmary
• paignton
• par
• sidmouth
• st-austell
• st-marychurch
• staustell
• stmarychurch
• teignmouth
• tiverton
• torquay
• totnes
• truro

Customers on these affected exchanges will have no ADSL connectivity, further updates will follow from our carrier when available.

A list of affected exchanges has been issued and is included below for reference.

UPDATE 14:51 A further 2 exchanges have also experienced the issue EGHAM and REDHILL , total connectivity loss will be experienced.

UPDATE 15:52 Engineers have now arrived at site to replace the affected supervisor card.

UPDATE 18:08 All affected exchanges are now back online, customers affected by this outage should be able to reconnect , if you cannot reconnect, please power off your router wait 2 minutes and power back on.

Our ADSL carrier has informed us that a number of exchanges (102) will experience intermittent connectivity issues while the DSLAM software is upgraded , this will allow us to expand the availability of our Premier Plus broadband package.

Users connected to the affected exchanges will be disconnected for upto 20 minutes while the software is upgraded, customers routers should reconnect automatically once the upgrade has been performed.