Hi,

The cause of state flaping ( Q->W->Q->W->Q  ) is the following i think:

* wn gets the job with stagein definition (set of files to be copied to WN)
* wn tries to copy the files so job turns into W state
* wn does several copy attempts before it gives up and returns job to a 
Q state (may take a long time)
* maui/moab tries to start the queued job again so the situation repeats 
until job gets held/canceled by scheduler. Some maui configurations put 
a job into a hold state after serveral unsuccesfull job start attempts. 
Such hold can be released automatically by maui after a certain period 
defined by DEFERTIME if i recall correctly

The fact is that job blocks resources and it must be deleted by hand






> Not sure, I only recently changed it on one of the CreamCEs and having just
> looked we have got jobs from that CreamCE in W state.
>
> I haven't checked if these were submitted before I made the change or
> whether they failed because the proxy was removed.
>
> I need to give the change a little longer then if there are still jobs
> appearing from the CE in W state do some more analysis.
>
> Yours,
> Chris.
>
>> -----Original Message-----
>> From: Massimo Sgaravatto [mailto:[log in to unmask]]
>> Sent: 22 February 2012 11:41
>> To: LHC Computer Grid - Rollout
>> Cc: Brew, Chris (STFC,RAL,PPD)
>> Subject: Re: [LCG-ROLLOUT] Torque/CREAM race condition at job
>> completion?
>>
>> On 02/22/2012 12:29 PM, Chris Brew wrote:
>>> Hi Jeff,
>>>
>>> I cannot take the credit, we've discussed this quite a bit in GridPP.
>>>
>>> There's at least one ticket in the system about it:
>>>
>>> https://ggus.eu/tech/ticket_show.php?ticket=72506
>>>
>>> I've recently tried setting 'delegation_purge_rate="-1"'[1] on one of
>> our
>>> CreamCEs to see if that helps. Initial indications are that it
>> doesn't.
>>
>>
>> You mean that the proxy is cleaned even when delegation_purge_rate is
>> -1, or you mean that the proxy is not cleaned but you still see the
>> Q->W->Q->  ... problem ?
>>
>>>
>>> Yours,
>>> Chris.
>>>
>>> [1]
>>>
>> http://grid.pd.infn.it/cream/field.php?n=Main.HowToConfigureTheProxyPur
>> ger
>>>
>>>> -----Original Message-----
>>>> From: LHC Computer Grid - Rollout [mailto:LCG-
>> [log in to unmask]]
>>>> On Behalf Of Jeff Templon
>>>> Sent: 22 February 2012 11:18
>>>> To: [log in to unmask]
>>>> Subject: Re: [LCG-ROLLOUT] Torque/CREAM race condition at job
>>>> completion?
>>>>
>>>> Hi,
>>>>
>>>> On Feb 22, 2012, at 11:29 , Chris Brew wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I believe this is what happens when a jobs proxy expires while the
>>>> job
>>>>> is in the queue.
>>>>>
>>>>> AFAICT, the Cream Proxy cleanup deletes the expired proxy from the
>>>>> cream proxy cache but does not cancel the job. So the job happily
>>>> sits
>>>>> in the queue until it torque tries to run it, at which point it
>> tries
>>>>> to stage in the proxy and of course fails.
>>>>>
>>>>> I think then it puts the job into waiting (W) state for a while
>>>> before
>>>>> requeuing it and it then just oscillates between Q and W until you
>>>>> delete it.
>>>>>
>>>>
>>>> We see exactly this : oscillations between state W and state Q,
>>>> continuing forever until the job is deleted.  Thanks for the
>>>> explanation.   Chris, you want to submit the GGUS ticket on this one
>> on
>>>> behalf of the rest of us?
>>>>
>>>> 				JT
>>>>
>>>>> It's annoying but you're not losing jobs that would have run.
>>>>>
>>>>> Yours,
>>>>> Chris.
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: LHC Computer Grid - Rollout [mailto:LCG-
>>>> [log in to unmask]]
>>>>>> On Behalf Of Jeff Templon
>>>>>> Sent: 22 February 2012 09:50
>>>>>> To: [log in to unmask]
>>>>>> Subject: Re: [LCG-ROLLOUT] Torque/CREAM race condition at job
>>>>>> completion?
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> we see these messages here too at Nikhef (giving up after 4
>>>> attempts)
>>>>>> and I cannot link them to jobs that have actually executed.  which
>>>> is
>>>>>> quite strange.
>>>>>>
>>>>>> this would indicate that somehow torque does not know that the job
>>>>>> has landed on a worker node, because there are no "start" or "end"
>>>>>> records that reference the WN in question in the torque server
>> logs.
>>>>>> however, the job has somehow landed on a worker node, because the
>>>>>> error message 'unable to copy file' is coming from the mom on a
>>>> worker node.
>>>>>>
>>>>>> This may be related to something we see here happen from time to
>>>> time.
>>>>>> A job is in state "Q", but somehow it has a worker node assigned
>> to
>>>> it.
>>>>>> We usually delete these jobs when we see them, as there is no way
>>>>>> we've found to 'unassign' the worker node.
>>>>>>
>>>>>>
>>>>> JT
>>>>>>
>>>>>> On Feb 22, 2012, at 09:20 , Gila Arrondo Miguel Angel wrote:
>>>>>>
>>>>>>> Hi Leslie,
>>>>>>>
>>>>>>> With a very similar setup (Moab/Torque on 1 host and CREAM-CE on
>>>>>> another) we've seen an error somehow close to what you describe
>>>> here.
>>>>>> In our /var/log/messages we find tons of entires like this:
>>>>>>>
>>>>>>> Feb 22 09:10:24 wn202 pbs_mom: LOG_ERROR::sys_copy, command
>>>>>>> '/usr/bin/scp -rpB err_cre02_780422084_StandardError
>>>>>>>
>>>>>>
>>>>
>> [log in to unmask]:/cream_localsandbox/data/atlas/_DC_ch_DC
>>>>>> _
>>>>>>>
>>>>>>
>>>>
>> cern_OU_Organic_Units_OU_Users_CN_atlpilo1_CN_614260_CN_Robot__ATLAS_
>>>>>> P
>>>>>>>
>>>>>>
>>>>
>> ilot1_atlas_Role_pilot_Capability_NULL_atlasplt/78/CREAM780422084/Sta
>>>>>> n
>>>>>>> dardError' failed with status=1, giving up after 4 attempts
>>>>>>>
>>>>>>> Feb 22 09:10:24 wn202 pbs_mom: LOG_ERROR::req_cpyfile, Unable to
>>>>>>> copy file err_cre02_780422084_StandardError to
>>>>>>>
>>>>>>
>>>>
>> [log in to unmask]:/cream_localsandbox/data/atlas/_DC_ch_DC
>>>>>> _
>>>>>>>
>>>>>>
>>>>
>> cern_OU_Organic_Units_OU_Users_CN_atlpilo1_CN_614260_CN_Robot__ATLAS_
>>>>>> P
>>>>>>>
>>>>>>
>>>>
>> ilot1_atlas_Role_pilot_Capability_NULL_atlasplt/78/CREAM780422084/Sta
>>>>>> n
>>>>>>> dardError
>>>>>>>
>>>>>>> Feb 22 09:10:29 wn202 pbs_mom: LOG_ERROR::sys_copy, command
>>>>>>> '/usr/bin/scp -rpB out_cre02_780422084_StandardOutput
>>>>>>>
>>>>>>
>>>>
>> [log in to unmask]:/cream_localsandbox/data/atlas/_DC_ch_DC
>>>>>> _
>>>>>>>
>>>>>>
>>>>
>> cern_OU_Organic_Units_OU_Users_CN_atlpilo1_CN_614260_CN_Robot__ATLAS_
>>>>>> P
>>>>>>>
>>>>>>
>>>>
>> ilot1_atlas_Role_pilot_Capability_NULL_atlasplt/78/CREAM780422084/Sta
>>>>>> n
>>>>>>> dardOutput' failed with status=1, giving up after 4 attempts F
>>>>>>>
>>>>>>> Have you checked whether these failed transfers are from
>> cancelled
>>>>>> jobs? In our experience, it was always the case.
>>>>>>>
>>>>>>> We've also looked at ways to mitigate this annoying verbosity,
>> but
>>>>>>> no
>>>>>> luck so far. The only option that we can think of is to stop using
>>>>>> scp for copies and move the sandbox to a shared area with the WNs,
>>>> so
>>>>>> you use regular cp ($usecp directive) and these errors are hidden.
>>>>>> But, of course, this approach has its own disadvantages as well.
>>>>>>>
>>>>>>> Does anyone else have a better idea?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Miguel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Miguel Gila
>>>>>>> CSCS Swiss National Supercomputing Centre HPC Solutions Via
>>>>>> Cantonale,
>>>>>>> Galleria 2 | CH-6928 Manno | Switzerland Miguel.gila [at] cscs.ch
>>>>>>>
>>>>>>> From: Leslie Groer
>>>>>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>>>>>> Reply-To: LHC Computer Grid - Rollout
>>>>>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>>>>>> Date: Thu, 16 Feb 2012 14:43:56 -0500
>>>>>>> To:<[log in to unmask]<mailto:LCG-
>> [log in to unmask]>>
>>>>>>> Subject: [LCG-ROLLOUT] Torque/CREAM race condition at job
>>>> completion?
>>>>>>>
>>>>>>> scp:
>>>>>>>
>>>>>>
>>>>
>> /opt/glite/var/cream_sandbox/atlasprd/_C_CA_O_Grid_OU_triumf_ca_CN_As
>>>>>> o
>>>>>>>
>>>>>>
>>>>
>> ka_De_Silva_GC1_atlas_Role_production_Capability_NULL_prdatl15/61/CRE
>>>>>> A
>>>>>>> M619112035/StandardOutput: No such file or directory
>>
>