Dear grid-users,
The (WCW-site-wide) cooling problem has now been resolved and power to the
nodes has been restored. Full service should resume within approx. 10 min.
Please report any remaining problems or inconsistencies to the
NDPF administrators at <[log in to unmask]>.
We apologize for the inconvenience caused.
Regards,
David Groep.
SYSTEMS AFFECTED
--------------------
node*-*.farmnet. LCG2ELPROD ALL WORKERNODES
ACTIONS
-----------
A fuse for the campus water cooling system was replaced in the
WCW campus ketelhuis and temperatures in the AMS-IX hot-spot areas
are now below 30C again (from >35C).
All nodes in nets 15, 16, and 17 have been restarted, old jobs drained
from the queues and the batch server and scheduler restarted.
Queues have been enabled again.
PROBLEMS
------------
Lack of incoming coolant water from the campus cental facilities:
On Mon, Jul 11, 2005 at 06:09:34PM +0200, David Groep wrote:
>
> SYSTEMS AFFECTED
> --------------------
> node*-*.farmnet LCG2ELPROD ALL WORKER NODES AT NIKHEF/NDPF
> *** NOTE NO SERVICE NODES ARE AFFECTED ***
>
> ACTIONS
> -----------
> Shut down all worker nodes and killed all jobs on the NIKHEF LCG2ELPROD
> system after a persistent cooling failure of a significant amount of
> cooling capacity. Jobs running (at the time from ATLAS and BIOMED) were
> lost.
> Queues have since been set to Draining, so no new jobs should get sent
> to the nikhef CE (tbn20.nikhef.nl).
>
> All service nodes, as well as all storage, REMAIN FULLY FUNCTIONAL.
> These service nodes are (at least for now) not affected by this problem.
>
> Appropriate downtime records will be entered everywhere soon.
>
> PROBLEMS
> ------------
> Ambient cooling systems failed and the repair did not finish within 60
> minutes. To prevent overheating of critical services, the "ordinary"
> worker nodes have been shut down to allow for the remaining cooling
> capacity to be used for those services.
>
>
> --
> David Groep
>
> ** National Institute for Nuclear and High Energy Physics, PDP projectgroup **
> ** Room: H1.57 Phone: +31 20 592 2179, PObox 41882, NL-1009 DB Amsterdam NL **
--
David Groep
** National Institute for Nuclear and High Energy Physics, PDP projectgroup **
** Room: H1.57 Phone: +31 20 592 2179, PObox 41882, NL-1009 DB Amsterdam NL **
|