Hi,
I am pleased to hear that these stress tests have shown that new BDII
seems to be even more stable that the old version. However, our own
testing has shown that there are one or two minor bugs with the version
that was used for these tests. Sites should NOT upgrade to this version
and should wait for the fully certified version that will be in the next
release.
Thanks
Laurence
Fotis Georgatos wrote:
> Dear LCG-rollout members,
>
>
>
> here follows a preliminary (but long!) report from the RB stress tests.
>
>
> First things first, a host running a BDII service should necessarily have
> the latest rpm, namely lcg-bdii-3.2.1-1; it resolves a number of race
> conditions, therefore it is much more stable and helps the respective RB
> become rock solid.
>
> During the submissions, I managed to confirm that the RBs have a breaking
> point at around 32-64 concurrent submissions of jobs and a throughput
> of 25
> jobs/minute. The exact threshold of submission failures might be
> dependent of
> the exact RB hardware but, there is one and it's possible to measure it
> reliably.
>
> That said, we have now our RB and BDII in Athens maintained at production
> level, feel free to use rb.isabella.grnet.gr and bdii.isabella.grnet.gr
> if you see fit.
>
>
> Let's dive now to individual job submissions, sites and queues...
>
> I sent a large number of tasks -a 5 second script- to 214 different
> queues.
> Out of a total of 3798 jobs, a 90% was a success, 5% was aborts, 5%
> other.
> After a couple of days of largely successful runs, here are the last
> pending
> jobs, their graphs' URLs, together with my observations/comments.
> If some admins agree that we should debug the issues, please contact me.
>
> [gef@ui01 8]$ cat allstatus5 |grep ^C|sort|uniq -c
> 205 Current Status: Aborted
> 3430 Current Status: Cleared
> 72 Current Status: Done (Failed)
> 47 Current Status: Ready
> 13 Current Status: Running
> 31 Current Status: Scheduled
>
> Analyze this.
>
>
> a) Scheduled
>
> [gef@ui01 8]$ cat allstatus5 |grep -2 '^C.*Scheduled'|grep ^D|cut -d:
> -f2|sort|uniq -c
> 14 grid01.phy.ncu.edu.tw
> 17 serv03.hep.phy.cam.ac.uk
>
> http://goc.grid.sinica.edu.tw/gstat/TW-NCUHEP/
> http://goc.grid.sinica.edu.tw/gstat/CAVENDISH-LCG2/
>
> These sites are excessively loaded, if compared to the few cpus they
> provide;
> This is typically a result of massive submissions that are done with
> the -r
> parameter, which is only a problem if people don't realise they can take
> advantage of the grid :-)
>
>
> b) Ready
>
> [gef@ui01 8]$ cat allstatus5 |grep -2 '^C.*Ready'|grep ^D|cut -d:
> -f2|sort|uniq -c
> 10 accip43.physik.rwth-aachen.de
> 10 bigmac-lcg-ce.physics.utoronto.ca
> 15 ce001.m45.ihep.su
> 8 ce.prd.hp.com
> 4 t2-ce-01.mi.infn.it
>
> http://goc.grid.sinica.edu.tw/gstat/RWTH-Aachen-Test/
> http://goc.grid.sinica.edu.tw/gstat/TORONTO-LCG2/ (Status:
> MAINTENANCE)
> http://goc.grid.sinica.edu.tw/gstat/RU-Protvino-IHEP/
> http://goc.grid.sinica.edu.tw/gstat/HPTC-LCG2/ (Status: NOTE?)
> http://goc.grid.sinica.edu.tw/gstat/INFN-MILANO-LCG2/ (Status:
> MAINTENANCE)
>
> The Aachen and Protvino sites seem to have the resources to execute
> the jobs,
> but for some reason they don't. The other sites seem to have non "OK"
> state,
> but I dunno what "NOTE" means at the HPTC-LCG2 site...
>
> In any case, these sites shouldn't appear in the result of an
> edg-job-list-match, otherwise there is an error, either typo or
> technical!
> (The last one indeed doesn't give a match any more but, what did you do?)
> (Yes, that's what they did in their queues: GlueCEStateStatus: Draining)
>
> Luckily, it is possible to retrieve a nearly meaningful Status Reason
> field:
>
> 10
> accip43.physik.rwth-aachen.de:2119/jobmanager-torque-dteam
> 47 the gatekeeper failed to run the job manager
> 10
> bigmac-lcg-ce.physics.utoronto.ca:2119/jobmanager-condor-dteam
> 93 the
> gatekeeper failed to find the requested service
> 3 ce001.m45.ihep.su:2119/jobmanager-pbs-dteam
> 76 cannot access cache files in ~/.globus/.gass_cache, check
> permissions,
> quota, and disk space
> 3 ce001.m45.ihep.su:2119/jobmanager-pbs-infinite
> 76 cannot access cache files in ~/.globus/.gass_cache, check
> permissions,
> quota, and disk space
> 3 ce001.m45.ihep.su:2119/jobmanager-pbs-long
> 76 cannot access cache files in ~/.globus/.gass_cache, check
> permissions,
> quota, and disk space
> 3 ce001.m45.ihep.su:2119/jobmanager-pbs-medium
> 76 cannot access cache files in ~/.globus/.gass_cache, check
> permissions,
> quota, and disk space
> 3 ce001.m45.ihep.su:2119/jobmanager-pbs-short
> 76 cannot access cache files in ~/.globus/.gass_cache, check
> permissions,
> quota, and disk space
> 2 ce.prd.hp.com:2119/jobmanager-pbs-dteam
> 93 the gatekeeper failed to find the requested service
> 6 ce.prd.hp.com:2119/jobmanager-torque-dteam
> 93 the gatekeeper failed to find the requested service
> 2 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-cert
> 3 an I/O operation failed
> 1 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-infinite
> 3 an I/O operation failed
> 1 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-short
> 3 an I/O operation failed
>
>
> c) Running
>
> [gef@ui01 8]$ cat allstatus5 |grep -2 '^C.*Running'|grep ^D|cut -d:
> -f2|sort|uniq -c
> 1 ce001.imbm.bas.bg
> 1 lcg06.gsi.de
> 1 pcncp04.ncp.edu.pk
> 10 wn-04-07-01-a.cr.cnaf.infn.it
>
> http://goc.grid.sinica.edu.tw/gstat/BG02-IM/
> http://goc.grid.sinica.edu.tw/gstat/GSI-LCG2/
> http://goc.grid.sinica.edu.tw/gstat/NCP-LCG2/ (Status: WARN)
> http://goc.grid.sinica.edu.tw/gstat/CNAF-T1/ (Status: ERROR)
>
> They are in Running state for 4 days which is weird. I'm sending
> JobIDs only
> for the first two, in case the relevant admins want to check with
> their CEs:
> https://rb.isabella.grnet.gr:9000/dIkVHUkEwN1fCbQwIsCy_Q
> https://rb.isabella.grnet.gr:9000/tTXjat-kWoY_aBA6eMUj5w
>
>
> d) Done (Failed)
>
> [gef@ui01 8]$ cat allstatus5 |grep -3 '^C.*Failed.'|grep ^D|cut -d:
> -f2|sort|uniq -c
> 1 cclcgceli01.in2p3.fr
> 7 ce1.egee.fr.cgg.com
> 15 ce.egee.man.poznan.pl
> 4 ce-iep-grid.saske.sk
> 8 ceitep.itep.ru
> 4 dgc-grid-35.brunel.ac.uk
> 1 grid003.ft.uam.es
> 1 lcg002.ihep.ac.cn
> 7 mallarme.cnb.uam.es
> 1 node03.datagrid.cea.fr
> 2 ramses.dsic.upv.es
> 13 t2-ce-01.mi.infn.it
> 1 t2ce01.physics.ox.ac.uk
> 7 testbed001.grid.ici.ro
>
> At least here it is possible to see the respective Status Reasons for the
> failures:
>
> 1 cclcgceli01.in2p3.fr:2119/jobmanager-bqs-G
> Got a job held event, reason: Globus error 24: the job manager
> detected an
> invalid script response
> 2 ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-long
> Got a job held event, reason: Unspecified gridmanager error
> 3 ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-medium
> Got a job held event, reason: Unspecified gridmanager error
> 2 ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-short
> Got a job held event, reason: Unspecified gridmanager error
> 6 ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-infinite
> Got a job held event, reason: Unspecified gridmanager error
> 4 ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-long
> Got a job held event, reason: Unspecified gridmanager error
> 5 ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-short
> Got a job held event, reason: Unspecified gridmanager error
> 4 ce-iep-grid.saske.sk:2119/jobmanager-torque-dteam
> Got a job held event, reason: Globus error 79: connecting to the job
> manager
> failed. Possible reasons: job terminated, invalid job contact, network
> problems, ...
> 8 ceitep.itep.ru:2119/jobmanager-torque-dteam
> Got a job held event, reason: Unspecified gridmanager error
> 4 dgc-grid-35.brunel.ac.uk:2119/jobmanager-lcgpbs-infinite
> Got a job held event, reason: Unspecified gridmanager error
> 1 grid003.ft.uam.es:2119/jobmanager-lcgpbs-long
> Got a job held event, reason: Unspecified gridmanager error
> 1 lcg002.ihep.ac.cn:2119/jobmanager-torque-dteam
> Got a job held event, reason: Unspecified gridmanager error
> 7 mallarme.cnb.uam.es:2119/jobmanager-pbs-lhcb
> Got a job held event, reason: Globus error 24: the job manager
> detected an
> invalid script response
> 1 node03.datagrid.cea.fr:2119/jobmanager-torque-dteam
> Got a job held event, reason: Unspecified gridmanager error
> 2 ramses.dsic.upv.es:2119/jobmanager-torque-dteam
> Got a job held event, reason: Unspecified gridmanager error
> 3 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-cert
> Got a job held event, reason: Globus error 3: an I/O operation failed
> 3 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-infinite
> Got a job held event, reason: Globus error 3: an I/O operation failed
> 4 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-long
> Got a job held event, reason: Globus error 3: an I/O operation failed
> 3 t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-short
> Got a job held event, reason: Globus error 3: an I/O operation failed
> 1 t2ce01.physics.ox.ac.uk:2119/jobmanager-lcgpbs-infinite
> Got a job held event, reason: Globus error 79: connecting to the job
> manager
> failed. Possible reasons: job terminated, invalid job contact, network
> problems, ...
> 7 testbed001.grid.ici.ro:2119/jobmanager-torque-dteam
> Got a job held event, reason: Unspecified gridmanager error
>
>
> e) Aborted
>
> So, here comes the interesting part... the set of jobs in Aborted state.
>
> [gef@ui01 8]$ cat allstatus5 |grep -2 '^C.*Aborted'|grep ^D|cut -d:
> -f2|sort|uniq -c
> 12 accip43.physik.rwth-aachen.de (*)
> 11 bigmac-lcg-ce.physics.utoronto.ca (*)
> 25 ce001.m45.ihep.su (*)
> 16 ce1.egee.fr.cgg.com
> 14 ce.egee.man.poznan.pl
> 5 ce-iep-grid.saske.sk
> 11 ceitep.itep.ru
> 10 ce.prd.hp.com (*)
> 7 dgc-grid-35.brunel.ac.uk
> 53 hotdog46.fnal.gov
> 7 lcg002.ihep.ac.cn
> 9 mallarme.cnb.uam.es
> 6 ramses.dsic.upv.es
> 17 t2-ce-01.mi.infn.it (*)
> 2 testbed001.grid.ici.ro
>
> The ones marked with (*) are sites with queues that have also tasks in
> the
> Ready state, so there could be a correlation, and this is a just another
> symptom of the same error. In fact, trying to telnet from our resource
> broker
> to port 2119 of some of these system, a firewall rule seem to apply,
> namely
> ceitep.itep.ru and dgc-grid-35.brunel.ac.uk.
>
> Also, a good reason for a job in the Aborted state, might be a temporary
> network failure, so I checked that they were available right now and
> did a
> rerun of 1 job per queue:
>
> Indeed,
> Destination: testbed001.grid.ici.ro:2119/jobmanager-torque-dteam
> Destination: ce.prd.hp.com:2119/jobmanager-torque-dteam
> Destination: ce-iep-grid.saske.sk:2119/jobmanager-torque-dteam
> might have been this case and just finished gracefully a few minutes ago.
>
> HPTC-LCG2 site was in NOTE Status, anyhow this is what happened with
> its second queue, it's hard to believe they both run pbs and torque:
>
> *************************************************************
> BOOKKEEPING INFORMATION:
>
> Status info for the Job :
> https://rb.isabella.grnet.gr:9000/SYvp9eRNttN_DPQn-NiuzQ
> Current Status: Aborted
> Status Reason: Job RetryCount (3) hit
> Destination: ce.prd.hp.com:2119/jobmanager-pbs-dteam
> reached on: Sun Mar 20 21:10:15 2005
> *************************************************************
>
>
> Another couple stay in the Ready state, and they are the 2 firewalled (?)
> systems:
> * ceitep.itep.ru
> * dgc-grid-35.brunel.ac.uk
>
> I believe these sites should either claim maintenance status,
> or fix the filtering rules or, suggest a better proposal ;-)
>
> The rest of sites went quickly in Aborted states and their list now is:
> Destination:
> accip43.physik.rwth-aachen.de:2119/jobmanager-torque-dteam
> Destination:
> bigmac-lcg-ce.physics.utoronto.ca:2119/jobmanager-condor-dteam
> Destination: ce001.m45.ihep.su:2119/jobmanager-pbs-dteam
> Destination: ce001.m45.ihep.su:2119/jobmanager-pbs-infinite
> Destination: ce001.m45.ihep.su:2119/jobmanager-pbs-long
> Destination: ce001.m45.ihep.su:2119/jobmanager-pbs-medium
> Destination: ce001.m45.ihep.su:2119/jobmanager-pbs-short
> Destination: ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-long
> Destination: ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-medium
> Destination: ce1.egee.fr.cgg.com:2119/jobmanager-lcgpbs-short
> Destination: ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-infinite
> Destination: ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-long
> Destination: ce.egee.man.poznan.pl:2119/jobmanager-lcgpbs-short
> Destination: ce.prd.hp.com:2119/jobmanager-pbs-dteam
> Destination:
> dgc-grid-35.brunel.ac.uk:2119/jobmanager-lcgpbs-infinite
> Destination: hotdog46.fnal.gov:2119/jobmanager-pbs-infinite
> Destination: hotdog46.fnal.gov:2119/jobmanager-pbs-long
> Destination: hotdog46.fnal.gov:2119/jobmanager-pbs-short
> Destination: lcg002.ihep.ac.cn:2119/jobmanager-torque-dteam
> Destination: mallarme.cnb.uam.es:2119/jobmanager-pbs-lhcb
> Destination: ramses.dsic.upv.es:2119/jobmanager-torque-dteam
> Destination: t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-cert
> Destination: t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-infinite
> Destination: t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-long
> Destination: t2-ce-01.mi.infn.it:2119/jobmanager-lcgpbs-short
>
> All of these tasks failed with "Job RetryCount (3) hit" error.
>
> Let me repeat this: if you want to debug stuff, contact me.
> I might not be quick but will definatelly reply at some point.
>
> If you managed to reach this far, you are brave lcg warriors
> and I wish you the best of luck with your next endeavours ;-)
>
> cheers,
> Fotis
>
> --
> echo "sysadmin know better bash than english" | sed s/min/mins/ \
> | sed 's/better bash/bash better/' # Yelling in a CERN forum
>
|