We lost power lunch time to the UPS room, some kind of overvolts which blew various trips, fuses, pdus and some devices. Smoke etc. HPD room with the CPUs and disk continued running and LPD room partially disrupted (as UPS supplies some racks) followed by full switch off to assess situation.
Estates re-established UPS power sometime around 4pm and we have begun re-instating power to the racks. Unfortunately we have definitely suffered damage to some equipment but as yet the extent is not clear. Main core switch (C300 for example lost 3 of 6 power supplies). We lost a full rack of 6 nortel switches in the LPD room. Tape drives looked OK but one of the controller boards blown and engineer attending. No idea yet about servers but one of our sister services lost 2 out of 10 power supplies.
Main UPS room cooling units both down - engineer called. We've decided to draw the line at re-establishing the network tonight (as this is already a sizable challenge) and tomorrow will focus on getting enough core up to get the FTS and BDIIs as soon as possible.
We will provide a briefing at the 13:30 experiment Liaison meeting assuming we can establish a connection.
Regards
Andrew
________________________________________
From: Testbed Support for GridPP member institutes [[log in to unmask]] on behalf of Linda Cornwall [[log in to unmask]]
Sent: Tuesday, November 20, 2012 2:26 PM
To: [log in to unmask]
Subject: Re: RAL network down
We have only just got e-mail back here at RAL - after being down for nearly 3 hours.
(Unofficially, and don't quote me. But from a conversation I heard in the corridor I think Tier 1 is also down due to a power failure.)
Linda.
> -----Original Message-----
> From: Testbed Support for GridPP member institutes [mailto:TB-
> [log in to unmask]] On Behalf Of Peter Gronbech
> Sent: 20 November 2012 13:55
> To: [log in to unmask]
> Subject: Re: RAL network down
>
> Two broadcasts have gone out so most sites should have heard by now.
> Pete
>
> --
> ----------------------------------------------------------------------
> Peter Gronbech Senior Systems Manager and Tel No. : 01865 273389
> GridPP Project Manager Fax No. : 01865 273418
> Department of Particle Physics,
> University of Oxford,
> Keble Road, Oxford OX1 3RH, UK E-mail : [log in to unmask]
> ----------------------------------------------------------------------
>
>
> -----Original Message-----
> From: Testbed Support for GridPP member institutes [mailto:TB-
> [log in to unmask]] On Behalf Of John Kewley
> Sent: 20 November 2012 13:53
> To: [log in to unmask]
> Subject: RAL network down
>
> There is a problem at RAL and so they have lost network connections.
>
> This affects a variety of services.
>
> I have no idea how long it'll take to fix, but it sounds like an electrical "failure"
> from what I hear so I suspect it'll be down for a bit.
>
> I thought I'd let you know since I hadn't seen anything on this list.
>
> Jens/Mike J: can one of you let igtf-general know (if you haven't already) that
> the CRLs are down and probably will for some time.
>
> Cheers
>
> JK
--
Scanned by iCritical.
|