I want to let our customers know about our progress in dealing with the service disruption that has been experienced since Monday.  This disruption has been caused by issues between VMWare ESX and our iSCSI storage, and we’ve taken the following steps to reduce and stop the service disruption.

• We have upgraded the RAM and firmware inside our storage unit called ‘iSCSI-2′, and it is now back in service.  We are currently moving some services across from iSCSI-3, and once we have reduced the load on iSCSI-3 enough, there should be no more disruption.  (The root cause of the disruption is trying to do too much with iSCSI-3, because we lost iSCSI-2 for a while).
• We have resolved the other configuration issues with our ESX cluster.
• We were unable to install additional storage on Wednesday.  Unfortunately, the hardware that we were reprovisioning didn’t work reliably enough during our testing; we’ve had to abandon this plan for now.
• Due to stock shortages, it will be Wednesday 28th Jan before we take delivery of the 4 x HP DL160 servers we’ve ordered to use as even more additional storage.  We’re aiming to install these a week on Sunday (Sun 1st Feb).

We will keep you informed as we make more progress with building and installing the additional storage.

We are conducting a lessons learned exercise early next week, and we will publish the results of this on here once that exercise is complete.

Related Posts

• No Related Post