Hi,
As you are no doubt aware yesterdays planned intervention to the RAL Tier1 to restructure the core networking within the Tier1 overran badly. Services were re-established (and the outage ended in the GOC DB) around 9pm local time (=UTC). We apologise for this significant overrun. Checking round our services this morning shows them all to be running OK.
The main problem during the change was to re-establish the paired 10Gbit link that provides the data path from the RAL Tier1 to the rest of the world. In order to get services back we are now running this link as a single connection and our maximum bandwith for data is limited to 10Gbit in each direction (as compared to the 20Gbit before). However, the network monitoring shows the data rate rarely exceeded 10Gbit on this link and consequently we don't expect the short term impact to be great. We are reviewing yesterday's events and will be preparing for a further intervention as soon as feasible to restore the uplink bandwidth. We do not expect to be able to do this this week.
Other aspects of yesterday's work were successful and enable a number of further changes to be made in the near future. This includes bringing this year's new capacity (cpu & disk) into use.
Gareth Smith
|