-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello
I have a number of lhcb jobs stuck in wait state, these jobs are trying
to run on several worker nodes. We have a shared file system and each
machine is able to mount the directories. I am getting the following
error via e-mail and have been since 09:50 yesterday. I also have a
number of globus-job-manager processes running on the CE (see below). I
have restarted pbs, maui and globus on the ce and I can ssh to the CE
from a worker node as lhcb001
PBS Job Id: 24610.helmsley.dur.scotgrid.ac.uk
Job Name: STDIN
File stage in failed, see below.
Job will be retried later, please investigate and correct problem.
Post job file processing error; job 24610.helmsley.dur.scotgrid.ac.uk on
host wn07.dur.scotgrid.ac.uk/1
Unable to copy file 24610.helms.OU to
helmsley.dur.scotgrid.ac.uk:/mt/home/lhcb001/.lcgjm/globus-cache-export.rhnqO5/batch.out
>>>>>> error from copy
helmsley.dur.scotgrid.ac.uk: Connection refused
.rhnqO5/batch.out: No such file or directory
>>>>>> end error output
Output retained on that host in: /var/spool/pbs/undelivered/24610.helms.OU
Unable to copy file 24610.helms.ER to
helmsley.dur.scotgrid.ac.uk:/mt/home/lhcb001/.lcgjm/globus-cache-export.rhnqO5/batch.err
>>>>>> error from copy
helmsley.dur.scotgrid.ac.uk: Connection refused
.rhnqO5/batch.err: No such file or directory
>>>>>> end error output
Output retained on that host in: /var/spool/pbs/undelivered/24610.helms.ER
-
----------------------------------------------------------------------------------------
lhcb001 12712 0.0 0.1 5520 3556 ? S Sep04 0:01
globus-job-manager -conf /opt/globus/etc/globus-job-manager.conf -type
fork -rdn jobmanager-fork -machine-type unknown -publish-jobs
lhcb001 20695 0.0 0.1 5516 3552 ? S Sep04 0:01
globus-job-manager -conf /opt/globus/etc/globus-job-manager.conf -type
fork -rdn jobmanager-fork -machine-type unknown -publish-jobs
lhcb001 28818 0.0 0.1 5516 3548 ? S Sep04 0:00
globus-job-manager -conf /opt/globus/etc/globus-job-manager.conf -type
fork -rdn jobmanager-fork -machine-type unknown -publish-jobs
lhcb001 29193 0.0 0.1 5520 3556 ? S Sep04 0:00
globus-job-manager -conf /opt/globus/etc/globus-job-manager.conf -type
fork -rdn jobmanager-fork -machine-type unknown -publish-jobs
lhcb001 29198 0.0 0.1 5516 3548 ? S Sep04 0:00
globus-job-manager -conf /opt/globus/etc/globus-job-manager.conf -type
fork -rdn jobmanager-fork -machine-type unknown -publish-jobs
lhcb001 12973 0.0 0.1 5516 3544 ? S 10:44 0:00
globus-job-manager -conf /opt/globus/etc/globus-job-manager.conf -type
fork -rdn jobmanager-fork -machine-type unknown -publish-jobs
lhcb001 13002 0.0 0.1 4348 2796 ? S 10:44 0:00 perl
/mt/home/lhcb001/.globus/.gass_cache/local/md5/4a/7df403a165c3ad81cfa6f459c5ae23/md5/82/0b2913e8e51abb8bea5d721ae8c439/data
-
--dest-url=https://lxn1177.cern.ch:20106/tmp/condor_g_scratch.0xab6e928.1638/helmsley.dur.scotgrid.ac.uk:2119.0x959bbb8/grid-mon
lhcb001 13245 0.0 0.3 8052 6516 ? S 10:45 0:00 perl
/tmp/grid_manager_monitor_agent.lhcb001.13002.1000 --delete-self
- --maxtime=3540s
lhcb001 13397 0.0 0.1 5520 3548 ? S 10:45 0:00
globus-job-manager -conf /opt/globus/etc/globus-job-manager.conf -type
fork -rdn jobmanager-fork -machine-type unknown -publish-jobs
lhcb001 13428 0.0 0.1 4320 2772 ? S 10:45 0:00 perl
/mt/home/lhcb001/.globus/.gass_cache/local/md5/27/01d6922efb71f243a80db333e4afe4/md5/9f/3ca83510506bf23ce4307675b8c52c/data
-
--dest-url=https://gdrb07.cern.ch:20001/tmp/condor_g_scratch.0x86bd2e8.31353/helmsley.dur.scotgrid.ac.uk:2119.0x9daf520/grid-mon
lhcb001 31300 0.0 0.3 8052 6528 ? S 11:38 0:00 perl
/tmp/grid_manager_monitor_agent.lhcb001.13002.1000 --delete-self
- --maxtime=3540s
root 31724 0.0 0.0 4760 672 pts/2 S 11:40 0:00 grep lhcb
[root@helmsley root]# ps axuw |grep
- --
- -------------------------------------------------------------
Mark Nelson - [log in to unmask]
IPPP, Department of Physics, University of Durham,
Science Laboratories, South Road, Durham, DH1 3LE
Office: +44 (0)191 334 3811, Direct Dial: +44 (0)191 334 3653
PGP Key: http://www.ippp.dur.ac.uk/~mn/pgp_key.txt
This mail is for the addressee only
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org
iD8DBQFDHCFNlzM++u0MgcERApEuAJ945nLgL++bG5xTDPUefvqsBEdpnwCfZ+ay
C7d7Gg3W2Ia3xLKnNLNCbk0=
=ZqiY
-----END PGP SIGNATURE-----
|