 |
 |
Hi, We here at DESY-HH saw jobs ending up in 'W' which were not able to retrieve from the CREAM-CE via scp parts of the ISB or in many cases the proxy in /var/glite/var/cream_sandbox/.../proxy/... Those jobs are lost forever but tend to stay in the system (as W). Maybe those jobs were removed by the WMS but not cancelled from the batch system by the CREAM-CE? Cheers Andreas On Wed, 22 Feb 2012, Chris Brew wrote: > Not sure, I only recently changed it on one of the CreamCEs and having just > looked we have got jobs from that CreamCE in W state. > > I haven't checked if these were submitted before I made the change or > whether they failed because the proxy was removed. > > I need to give the change a little longer then if there are still jobs > appearing from the CE in W state do some more analysis. > > Yours, > Chris. > >> -----Original Message----- >> From: Massimo Sgaravatto [mailto:[log in to unmask]] >> Sent: 22 February 2012 11:41 >> To: LHC Computer Grid - Rollout >> Cc: Brew, Chris (STFC,RAL,PPD) >> Subject: Re: [LCG-ROLLOUT] Torque/CREAM race condition at job >> completion? >> >> On 02/22/2012 12:29 PM, Chris Brew wrote: >>> Hi Jeff, >>> >>> I cannot take the credit, we've discussed this quite a bit in GridPP. >>> >>> There's at least one ticket in the system about it: >>> >>> https://ggus.eu/tech/ticket_show.php?ticket=72506 >>> >>> I've recently tried setting 'delegation_purge_rate="-1"'[1] on one of >> our >>> CreamCEs to see if that helps. Initial indications are that it >> doesn't. >> >> >> You mean that the proxy is cleaned even when delegation_purge_rate is >> -1, or you mean that the proxy is not cleaned but you still see the >> Q->W->Q-> ... problem ? >> >>> >>> Yours, >>> Chris. >>> >>> [1] >>> >> http://grid.pd.infn.it/cream/field.php?n=Main.HowToConfigureTheProxyPur >> ger >>> >>>> -----Original Message----- >>>> From: LHC Computer Grid - Rollout [mailto:LCG- >> [log in to unmask]] >>>> On Behalf Of Jeff Templon >>>> Sent: 22 February 2012 11:18 >>>> To: [log in to unmask] >>>> Subject: Re: [LCG-ROLLOUT] Torque/CREAM race condition at job >>>> completion? >>>> >>>> Hi, >>>> >>>> On Feb 22, 2012, at 11:29 , Chris Brew wrote: >>>> >>>>> Hi, >>>>> >>>>> I believe this is what happens when a jobs proxy expires while the >>>> job >>>>> is in the queue. >>>>> >>>>> AFAICT, the Cream Proxy cleanup deletes the expired proxy from the >>>>> cream proxy cache but does not cancel the job. So the job happily >>>> sits >>>>> in the queue until it torque tries to run it, at which point it >> tries >>>>> to stage in the proxy and of course fails. >>>>> >>>>> I think then it puts the job into waiting (W) state for a while >>>> before >>>>> requeuing it and it then just oscillates between Q and W until you >>>>> delete it. >>>>> >>>> >>>> We see exactly this : oscillations between state W and state Q, >>>> continuing forever until the job is deleted. Thanks for the >>>> explanation. Chris, you want to submit the GGUS ticket on this one >> on >>>> behalf of the rest of us? >>>> >>>> JT >>>> >>>>> It's annoying but you're not losing jobs that would have run. >>>>> >>>>> Yours, >>>>> Chris. >>>>> >>>>>> -----Original Message----- >>>>>> From: LHC Computer Grid - Rollout [mailto:LCG- >>>> [log in to unmask]] >>>>>> On Behalf Of Jeff Templon >>>>>> Sent: 22 February 2012 09:50 >>>>>> To: [log in to unmask] >>>>>> Subject: Re: [LCG-ROLLOUT] Torque/CREAM race condition at job >>>>>> completion? >>>>>> >>>>>> Hi, >>>>>> >>>>>> we see these messages here too at Nikhef (giving up after 4 >>>> attempts) >>>>>> and I cannot link them to jobs that have actually executed. which >>>> is >>>>>> quite strange. >>>>>> >>>>>> this would indicate that somehow torque does not know that the job >>>>>> has landed on a worker node, because there are no "start" or "end" >>>>>> records that reference the WN in question in the torque server >> logs. >>>>>> however, the job has somehow landed on a worker node, because the >>>>>> error message 'unable to copy file' is coming from the mom on a >>>> worker node. >>>>>> >>>>>> This may be related to something we see here happen from time to >>>> time. >>>>>> A job is in state "Q", but somehow it has a worker node assigned >> to >>>> it. >>>>>> We usually delete these jobs when we see them, as there is no way >>>>>> we've found to 'unassign' the worker node. >>>>>> >>>>>> >>>>> JT >>>>>> >>>>>> On Feb 22, 2012, at 09:20 , Gila Arrondo Miguel Angel wrote: >>>>>> >>>>>>> Hi Leslie, >>>>>>> >>>>>>> With a very similar setup (Moab/Torque on 1 host and CREAM-CE on >>>>>> another) we've seen an error somehow close to what you describe >>>> here. >>>>>> In our /var/log/messages we find tons of entires like this: >>>>>>> >>>>>>> Feb 22 09:10:24 wn202 pbs_mom: LOG_ERROR::sys_copy, command >>>>>>> '/usr/bin/scp -rpB err_cre02_780422084_StandardError >>>>>>> >>>>>> >>>> >> [log in to unmask]:/cream_localsandbox/data/atlas/_DC_ch_DC >>>>>> _ >>>>>>> >>>>>> >>>> >> cern_OU_Organic_Units_OU_Users_CN_atlpilo1_CN_614260_CN_Robot__ATLAS_ >>>>>> P >>>>>>> >>>>>> >>>> >> ilot1_atlas_Role_pilot_Capability_NULL_atlasplt/78/CREAM780422084/Sta >>>>>> n >>>>>>> dardError' failed with status=1, giving up after 4 attempts >>>>>>> >>>>>>> Feb 22 09:10:24 wn202 pbs_mom: LOG_ERROR::req_cpyfile, Unable to >>>>>>> copy file err_cre02_780422084_StandardError to >>>>>>> >>>>>> >>>> >> [log in to unmask]:/cream_localsandbox/data/atlas/_DC_ch_DC >>>>>> _ >>>>>>> >>>>>> >>>> >> cern_OU_Organic_Units_OU_Users_CN_atlpilo1_CN_614260_CN_Robot__ATLAS_ >>>>>> P >>>>>>> >>>>>> >>>> >> ilot1_atlas_Role_pilot_Capability_NULL_atlasplt/78/CREAM780422084/Sta >>>>>> n >>>>>>> dardError >>>>>>> >>>>>>> Feb 22 09:10:29 wn202 pbs_mom: LOG_ERROR::sys_copy, command >>>>>>> '/usr/bin/scp -rpB out_cre02_780422084_StandardOutput >>>>>>> >>>>>> >>>> >> [log in to unmask]:/cream_localsandbox/data/atlas/_DC_ch_DC >>>>>> _ >>>>>>> >>>>>> >>>> >> cern_OU_Organic_Units_OU_Users_CN_atlpilo1_CN_614260_CN_Robot__ATLAS_ >>>>>> P >>>>>>> >>>>>> >>>> >> ilot1_atlas_Role_pilot_Capability_NULL_atlasplt/78/CREAM780422084/Sta >>>>>> n >>>>>>> dardOutput' failed with status=1, giving up after 4 attempts F >>>>>>> >>>>>>> Have you checked whether these failed transfers are from >> cancelled >>>>>> jobs? In our experience, it was always the case. >>>>>>> >>>>>>> We've also looked at ways to mitigate this annoying verbosity, >> but >>>>>>> no >>>>>> luck so far. The only option that we can think of is to stop using >>>>>> scp for copies and move the sandbox to a shared area with the WNs, >>>> so >>>>>> you use regular cp ($usecp directive) and these errors are hidden. >>>>>> But, of course, this approach has its own disadvantages as well. >>>>>>> >>>>>>> Does anyone else have a better idea? >>>>>>> >>>>>>> Cheers, >>>>>>> Miguel >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Miguel Gila >>>>>>> CSCS Swiss National Supercomputing Centre HPC Solutions Via >>>>>> Cantonale, >>>>>>> Galleria 2 | CH-6928 Manno | Switzerland Miguel.gila [at] cscs.ch >>>>>>> >>>>>>> From: Leslie Groer >>>>>>> <[log in to unmask]<mailto:[log in to unmask]>> >>>>>>> Reply-To: LHC Computer Grid - Rollout >>>>>>> <[log in to unmask]<mailto:[log in to unmask]>> >>>>>>> Date: Thu, 16 Feb 2012 14:43:56 -0500 >>>>>>> To:<[log in to unmask]<mailto:LCG- >> [log in to unmask]>> >>>>>>> Subject: [LCG-ROLLOUT] Torque/CREAM race condition at job >>>> completion? >>>>>>> >>>>>>> scp: >>>>>>> >>>>>> >>>> >> /opt/glite/var/cream_sandbox/atlasprd/_C_CA_O_Grid_OU_triumf_ca_CN_As >>>>>> o >>>>>>> >>>>>> >>>> >> ka_De_Silva_GC1_atlas_Role_production_Capability_NULL_prdatl15/61/CRE >>>>>> A >>>>>>> M619112035/StandardOutput: No such file or directory >> > > # Andreas Gellrich # DESY IT / Grid Computing # 2b/317, Notkestr. 85, D-22607 Hamburg, +49 40 8998 2732