Several LLU exchanges connected to the Shareshill aggregation router have lost connectivity due to a backhaul fault, customers connected to these exchanges will not have any DSL connection available.
All exchanges will still have 999 emergency call functionality.
BT are currently investigating. Exchanges affected have been listed below.
Broadwell, Great Barr, Stourbridge, Kingswinford, Kidderminster, Tipton and Dudley.
UPDATE 13:34 - We have been informed that BT engineers are onsite carrying out fibre repairs , further updates will follow.
UPDATE 15:09 - BT Engineers are continuing to repair the fibre that has become damaged.
UPDATE 09:12 25/2/10 - Services have now been restored to all exchanges above apart from Tipton, BT engineering are continuing to investigate and repair fibre.
UPDATE 09:02 26/2/10 - Services to all exchanges listed above have now been restored.
We have discovered that Microsoft Hotmail/Yahoo/Live networks have decided to block our mailservers from sending outbound mail directly to their MX servers. We have requested removal from this blocklist and are awaiting Microsoft to action.
We apologise for any inconvenience this causes and would like to remind anyone using us to forward mail to one of these accounts to not ‘report as spam’ if the mail is only forwarded through us. This blacklists our network, not the originating network.
***Update 10:10***
We are now only subject to the normal Microsoft ‘throttling’ for all of their inbound servers so the vast majority of emails should now be reaching their intended target. We apologise for any problems resulting from this and we have asked for our sending limits to be raised across all Microsoft servers.
We are investigating a problem with the web cluster following last night’s maintenance. Status will be updated here as soon as further information is available. We apologise for any inconvenience caused.
*** Update 07:50 ***
This problem was inadvertently caused by some emergency security changes made last night. Those changes have now been reverted, pending review and the web cluster has been re-synchronised. All PHP 4 and 5.2 sites are now back online.
We will be switching off parts of our core network in the very early hours of Friday morning to perform some upgrades and to increase power resiliency. Some parts of our network will be unavailable for short periods of time between 00:00 and 05:00. During this period all phones will re-register at some point, some websites will be offline and all Gradwell DSL lines will lose connection for up to 20 minutes.
Whilst the whole network will be ‘at risk’ for the entire period, the main windows for major network outages are specific per device. Our work plan will be as follows:-
Core Switching - 01:00-2:00
ADSL Routers - 02:00-02:30
Border Routers - 02:30-03:30
Peering Routers - 03:30-04:00
We apologise in advance for any loss of services caused by this and if you experience any problems after this work has been completed, please contact support@gradwell.com
*** UPDATE 07:05 ***
We are investigating a problem with the web cluster following this maintenance. Status will be updated here as soon as further information is available. We apologise for any inconvenience caused.
***Update 13:38***
We have been monitoring all upgrades and they are performing as expected. All work was successfully completed. We apologise again for any loss of service during these upgrades.
Some users who are connecting via our NAT proxy are seeing some registration issues. Our server admin team are working on this now and we will post an update as soon as possible.
We apologise for any problems this may be causing you
***Update 12:30***
Our system admin team have fully restored service and all of our NAT proxies are fully functioning. We again apologise for any problems this has caused.
We are currently seeing a massive influx of mail through our inbound mail servers and this is causing delays on mail delivery across the network. Our system admin team are working on this now and we will update here with further details as soon as possible. We apologise for any problems caused by this.
UPDATE: All mail queues on our systems have now cleared and mail delivery is proceeding as normal. Our investigations have highlighted a number of mail sources delivering abnormally large volumes of email whom we have taken action against. Further to this we have highlighted some architectural improvements within our own systems in order to mitigate these types of attacks in the future.
***Update 13:20 12/2***
We are once more seeing a massive amount of inbound mail causing some queues on our mail platform. Our system administration team are working on this now and hope to have any backlog cleared as soon as possible. Again we apologise for any problems caused by this and will update here again once the queues have been fully cleared
***Update 15:58***
Our queues have held at a low level for quite a while now but we are still seeing a huge amount of inbound connections to our edge nodes. Our system admin team are bringing extra resources online to help keep delays to a minimum. We are continuing to monitor the situation and will update here again.
***Update 13/02/10 13:00***
Systems are currently processing new mail well, however, we have identified two servers which have messages stuck in their local delivery queues from Friday. We are manually flushing those messages at the moment so customers may see some delayed messages arriving in their inbox.
***Update 15:40 16/2/10***
We have been closely monitoring the mail platform and the previous delays have been fully cleared and no further delays have been seen.
This was partially caused by header corruption breaking our mail loop detection coding. This has now been recoded and fully tested. We apologise again for any problems caused by these delays.
We will be performing some further maintenance to one of our hosted unified comms nodes tonight from now until 23:30. This might cause a few minutes downtime whilst we failover the primary node. We will post here again when the maintenance work is completed
We apologise for any problems this might cause you, but would like to assure all users that any downtime will be kept to a minimum and in most cases wont even be noticed due to our clustered server setup.
Users of other hosted VoIP products including Centrex/multi user/SIP/IAX trunks will be unaffected.
***Update 22:50***
We have finished the maintenance ahead of schedule and our hosted unified comms platform is now running in full high availability. We apologise if anyone experienced a dropped call due to this.
Starts: 2010/02/09 22:00 Ends 2010/02/09 23:00
On the evening of Tuesday 9th February we are performing some maintenance on our VoIP NAT proxy cluster to improve its resilience to hardware and software failures.
We do not expect there to be any significant downtime during this work, however if you do experience problems making or receiving calls, we recommend you reboot your phone in the first instance.
We apologize for any inconvenience this work may cause you.
Update (Wednesday, 10:23): This maintenance was completed last night with minimal downtime, and the NAT proxy is functioning normally today.
Some users may be seeing problems making or receiving calls using the Unified Comms platform. Our system admin team are looking at this now and we will post any updates here.
We apologise for any problems caused by this
***Update 13:55***
The Unified Comms platform is now running at 100% We apologise again for any problems caused.
We are performing some routine maintenance on our Unified Comms Platform at around 22:00 this evening. This will cause a downtime of approximately 20 minutes, during this time calls may not be routed correctly.We will update here when the work is started and again when the work is completed.
Users of our hosted VoIP platform, and SIP/IAX trunk users will be unaffected by this maintenance.
***Update 22:00***
We are about to start the planned maintenance on schedule. The Unified Comms platform is now offline
***Update 22:12***
We have finished the maintenance a little early. Our Unified Comms platform is now back up and running in full High Availability mode.
We apologise for any inconvenience caused by this maintenance.
