Dear LCG-rollout members,
here follows a preliminary (but long!) report from the RB stress tests.
First things first, a host running a BDII service should necessarily have
the latest rpm, namely lcg-bdii-3.2.1-1; it resolves a number of race
conditions, therefore it is much more stable and helps the respective RB
become rock solid.
During the submissions, I managed to confirm that the RBs have a breaking
point at around 32-64 concurrent submissions of jobs and a throughput of 25
jobs/minute. The exact threshold of submission failures might be dependent of
the exact RB hardware but, there is one and it's possible to measure it
reliably.
That said, we have now our RB and BDII in Athens maintained at production
level, feel free to use rb.isabella.grnet.gr and bdii.isabella.grnet.gr
if you see fit.
Let's dive now to individual job submissions, sites and queues...
I sent a large number of tasks -a 5 second script- to 214 different queues.
Out of a total of 3798 jobs, a 90% was a success, 5% was aborts, 5% other.
After a couple of days of largely successful runs, here are the last pending
jobs, their graphs' URLs, together with my observations/comments.
If some admins agree that we should debug the issues, please contact me.
[gef@ui01 8]$ cat allstatus5 |grep ^C|sort|uniq -c
205 Current Status: Aborted
3430 Current Status: Cleared
72 Current Status: Done (Failed)
47 Current Status: Ready
13 Current Status: Running
31 Current Status: Scheduled
Analyze this.
a) Scheduled
[gef@ui01 8]$ cat allstatus5 |grep -2 '^C.*Scheduled'|grep ^D|cut -d:
-f2|sort|uniq -c
14 grid01.phy.ncu.edu.tw
17 serv03.hep.phy.cam.ac.uk
http://goc.grid.sinica.edu.tw/gstat/TW-NCUHEP/
http://goc.grid.sinica.edu.tw/gstat/CAVENDISH-LCG2/
These sites are excessively loaded, if compared to the few cpus they provide;
This is typically a result of massive submissions that are done with the -r
parameter, which is only a problem if people don't realise they can take
advantage of the grid :-)
b) Ready
[gef@ui01 8]$ cat allstatus5 |grep -2 '^C.*Ready'|grep ^D|cut -d: -f2|sort|uniq -c
10 accip43.physik.rwth-aachen.de
10 bigmac-lcg-ce.physics.utoronto.ca
15 ce001.m45.ihep.su
8 ce.prd.hp.com
4 t2-ce-01.mi.infn.it
http://goc.grid.sinica.edu.tw/gstat/RWTH-Aachen-Test/
http://goc.grid.sinica.edu.tw/gstat/TORONTO-LCG2/ (Status: MAINTENANCE)
http://goc.grid.sinica.edu.tw/gstat/RU-Protvino-IHEP/
http://goc.grid.sinica.edu.tw/gstat/HPTC-LCG2/ (Status: NOTE?)
http://goc.grid.sinica.edu.tw/gstat/INFN-MILANO-LCG2/ (Status: MAINTENANCE)
The Aachen and Protvino sites seem to have the resources to execute the jobs,
but for some reason they don't. The other sites seem to have non "OK" state,
but I dunno what "NOTE" means at the HPTC-LCG2 site...
In any case, these sites shouldn't appear in the result of an
edg-job-list-match, otherwise there is an error, either typo or technical!
(The last one indeed doesn't give a match any more but, what did you do?)
(Yes, that's what they did in their queues: GlueCEStateStatus: Draining)
Luckily, it is possible to retrieve a nearly meaningful Status Reason field:
10 accip43.physik.rwth-aachen.de:2119/jobmanager-torque-dteam
47 the gatekeeper failed to run the job manager
10
bigmac-lcg-ce.physics.utoronto.ca:2119/jobmanager-condor-dteam 93 the
gatekeeper failed to find the requested service
3 ce001.m45.ihep.su:2119/jobmanager-pbs-dteam
76 cannot access cache files in ~/.globus/.gass_cache, check permissions,
quota, and disk space
3 ce001.m45.ihep.su:2119/jobmanager-pbs-infinite
76 cannot access cache files in ~/.globus/.gass_cache, check permissions,
quota, and disk space
3 ce001.m45.ihep.su:2119/jobmanager-pbs-long
76 cannot access cache files in ~/.globus/.gass_cache, check permissions,
quota, and disk space
3 ce001.m45.ihep.su:2119/jobmanager-pbs-medium
76 cannot access cache files in ~/.globus/.gass_cache, check permissions,
quota, and disk space
3 ce001.m45.ihep.su:2119/jobmanager-pbs-short
76 cannot access cache files in ~/.globus/.gass_cache, check permissions,
quota, and disk space
2 ce.prd.hp.com:2119/jobmanager-pbs-dteam
93 the gatekeeper failed to find the requested service
6 ce.prd.hp.com:2119/jobmanager-torque-dteam
93 the gatekeeper failed to find the requested service
2 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-cert
3 an I/O operation failed
1 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-infinite
3 an I/O operation failed
1 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-short
3 an I/O operation failed
c) Running
[gef@ui01 8]$ cat allstatus5 |grep -2 '^C.*Running'|grep ^D|cut -d:
-f2|sort|uniq -c
1 ce001.imbm.bas.bg
1 lcg06.gsi.de
1 pcncp04.ncp.edu.pk
10 wn-04-07-01-a.cr.cnaf.infn.it
http://goc.grid.sinica.edu.tw/gstat/BG02-IM/
http://goc.grid.sinica.edu.tw/gstat/GSI-LCG2/
http://goc.grid.sinica.edu.tw/gstat/NCP-LCG2/ (Status: WARN)
http://goc.grid.sinica.edu.tw/gstat/CNAF-T1/ (Status: ERROR)
They are in Running state for 4 days which is weird. I'm sending JobIDs only
for the first two, in case the relevant admins want to check with their CEs:
https://rb.isabella.grnet.gr:9000/dIkVHUkEwN1fCbQwIsCy_Q
https://rb.isabella.grnet.gr:9000/tTXjat-kWoY_aBA6eMUj5w
d) Done (Failed)
[gef@ui01 8]$ cat allstatus5 |grep -3 '^C.*Failed.'|grep ^D|cut -d:
-f2|sort|uniq -c
1 cclcgceli01.in2p3.fr
7 ce1.egee.fr.cgg.com
15 ce.egee.man.poznan.pl
4 ce-iep-grid.saske.sk
8 ceitep.itep.ru
4 dgc-grid-35.brunel.ac.uk
1 grid003.ft.uam.es
1 lcg002.ihep.ac.cn
7 mallarme.cnb.uam.es
1 node03.datagrid.cea.fr
2 ramses.dsic.upv.es
13 t2-ce-01.mi.infn.it
1 t2ce01.physics.ox.ac.uk
7 testbed001.grid.ici.ro
At least here it is possible to see the respective Status Reasons for the
failures:
1 cclcgceli01.in2p3.fr:2119/jobmanager-bqs-G
Got a job held event, reason: Globus error 24: the job manager detected an
invalid script response
2 ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-long
Got a job held event, reason: Unspecified gridmanager error
3 ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-medium
Got a job held event, reason: Unspecified gridmanager error
2 ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-short
Got a job held event, reason: Unspecified gridmanager error
6 ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-infinite
Got a job held event, reason: Unspecified gridmanager error
4 ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-long
Got a job held event, reason: Unspecified gridmanager error
5 ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-short
Got a job held event, reason: Unspecified gridmanager error
4 ce-iep-grid.saske.sk:2119/jobmanager-torque-dteam
Got a job held event, reason: Globus error 79: connecting to the job manager
failed. Possible reasons: job terminated, invalid job contact, network
problems, ...
8 ceitep.itep.ru:2119/jobmanager-torque-dteam
Got a job held event, reason: Unspecified gridmanager error
4 dgc-grid-35.brunel.ac.uk:2119/jobmanager-lcgpbs-infinite
Got a job held event, reason: Unspecified gridmanager error
1 grid003.ft.uam.es:2119/jobmanager-lcgpbs-long
Got a job held event, reason: Unspecified gridmanager error
1 lcg002.ihep.ac.cn:2119/jobmanager-torque-dteam
Got a job held event, reason: Unspecified gridmanager error
7 mallarme.cnb.uam.es:2119/jobmanager-pbs-lhcb
Got a job held event, reason: Globus error 24: the job manager detected an
invalid script response
1 node03.datagrid.cea.fr:2119/jobmanager-torque-dteam
Got a job held event, reason: Unspecified gridmanager error
2 ramses.dsic.upv.es:2119/jobmanager-torque-dteam
Got a job held event, reason: Unspecified gridmanager error
3 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-cert
Got a job held event, reason: Globus error 3: an I/O operation failed
3 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-infinite
Got a job held event, reason: Globus error 3: an I/O operation failed
4 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-long
Got a job held event, reason: Globus error 3: an I/O operation failed
3 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-short
Got a job held event, reason: Globus error 3: an I/O operation failed
1 t2ce01.physics.ox.ac.uk:2119/jobmanager-lcgpbs-infinite
Got a job held event, reason: Globus error 79: connecting to the job manager
failed. Possible reasons: job terminated, invalid job contact, network
problems, ...
7 testbed001.grid.ici.ro:2119/jobmanager-torque-dteam
Got a job held event, reason: Unspecified gridmanager error
e) Aborted
So, here comes the interesting part... the set of jobs in Aborted state.
[gef@ui01 8]$ cat allstatus5 |grep -2 '^C.*Aborted'|grep ^D|cut -d:
-f2|sort|uniq -c
12 accip43.physik.rwth-aachen.de (*)
11 bigmac-lcg-ce.physics.utoronto.ca (*)
25 ce001.m45.ihep.su (*)
16 ce1.egee.fr.cgg.com
14 ce.egee.man.poznan.pl
5 ce-iep-grid.saske.sk
11 ceitep.itep.ru
10 ce.prd.hp.com (*)
7 dgc-grid-35.brunel.ac.uk
53 hotdog46.fnal.gov
7 lcg002.ihep.ac.cn
9 mallarme.cnb.uam.es
6 ramses.dsic.upv.es
17 t2-ce-01.mi.infn.it (*)
2 testbed001.grid.ici.ro
The ones marked with (*) are sites with queues that have also tasks in the
Ready state, so there could be a correlation, and this is a just another
symptom of the same error. In fact, trying to telnet from our resource broker
to port 2119 of some of these system, a firewall rule seem to apply, namely
ceitep.itep.ru and dgc-grid-35.brunel.ac.uk.
Also, a good reason for a job in the Aborted state, might be a temporary
network failure, so I checked that they were available right now and did a
rerun of 1 job per queue:
Indeed,
Destination: testbed001.grid.ici.ro:2119/jobmanager-torque-dteam
Destination: ce.prd.hp.com:2119/jobmanager-torque-dteam
Destination: ce-iep-grid.saske.sk:2119/jobmanager-torque-dteam
might have been this case and just finished gracefully a few minutes ago.
HPTC-LCG2 site was in NOTE Status, anyhow this is what happened with
its second queue, it's hard to believe they both run pbs and torque:
*************************************************************
BOOKKEEPING INFORMATION:
Status info for the Job : https://rb.isabella.grnet.gr:9000/SYvp9eRNttN_DPQn-NiuzQ
Current Status: Aborted
Status Reason: Job RetryCount (3) hit
Destination: ce.prd.hp.com:2119/jobmanager-pbs-dteam
reached on: Sun Mar 20 21:10:15 2005
*************************************************************
Another couple stay in the Ready state, and they are the 2 firewalled (?)
systems:
* ceitep.itep.ru
* dgc-grid-35.brunel.ac.uk
I believe these sites should either claim maintenance status,
or fix the filtering rules or, suggest a better proposal ;-)
The rest of sites went quickly in Aborted states and their list now is:
Destination: accip43.physik.rwth-aachen.de:2119/jobmanager-torque-dteam
Destination: bigmac-lcg-ce.physics.utoronto.ca:2119/jobmanager-condor-dteam
Destination: ce001.m45.ihep.su:2119/jobmanager-pbs-dteam
Destination: ce001.m45.ihep.su:2119/jobmanager-pbs-infinite
Destination: ce001.m45.ihep.su:2119/jobmanager-pbs-long
Destination: ce001.m45.ihep.su:2119/jobmanager-pbs-medium
Destination: ce001.m45.ihep.su:2119/jobmanager-pbs-short
Destination: ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-long
Destination: ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-medium
Destination: ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-short
Destination: ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-infinite
Destination: ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-long
Destination: ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-short
Destination: ce.prd.hp.com:2119/jobmanager-pbs-dteam
Destination: dgc-grid-35.brunel.ac.uk:2119/jobmanager-lcgpbs-infinite
Destination: hotdog46.fnal.gov:2119/jobmanager-pbs-infinite
Destination: hotdog46.fnal.gov:2119/jobmanager-pbs-long
Destination: hotdog46.fnal.gov:2119/jobmanager-pbs-short
Destination: lcg002.ihep.ac.cn:2119/jobmanager-torque-dteam
Destination: mallarme.cnb.uam.es:2119/jobmanager-pbs-lhcb
Destination: ramses.dsic.upv.es:2119/jobmanager-torque-dteam
Destination: t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-cert
Destination: t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-infinite
Destination: t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-long
Destination: t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-short
All of these tasks failed with "Job RetryCount (3) hit" error.
Let me repeat this: if you want to debug stuff, contact me.
I might not be quick but will definatelly reply at some point.
If you managed to reach this far, you are brave lcg warriors
and I wish you the best of luck with your next endeavours ;-)
cheers,
Fotis
--
echo "sysadmin know better bash than english" | sed s/min/mins/ \
| sed 's/better bash/bash better/' # Yelling in a CERN forum
|
