Hi Chris
I haven't opened a ticket separately but ticket opened by Stephan explain it in detail (https://ggus.eu/tech/ticket_show.php?ticket=72506 )
It is not a proxy issue as such but it is a bug in cream that it does not handle jobs properly whose proxy expire while still in queue. As for I understand if you upload a long proxy to myproxy server and then submit a job with initial default 12 hour proxy to CE and job gets chance to run within 12 hour then CE will keep updating the proxy unless job finishes. But if it couldn't get chance to run with in 12 hour then there is no mechanism to update proxy. Ideally cream should handle this and cancel the job. But somehow cream delete expired delegation without checking that there is job in the queue. There is a savannah ticket for this https://savannah.cern.ch/bugs/?86700.
I am just going to delete 70 esr jobs because they are in this state. It seems that a esr user submitted a bunch of job to Oxford and first batch of jobs ran successfully but as they crossed their fair share, rest of jobs got lower priority and it went into this weird state.
Cheers
Kashif
-----Original Message-----
From: Christopher J. Walker [mailto:[log in to unmask]]
Sent: 13 June 2012 22:28
To: Testbed Support for GridPP member institutes
Cc: Kashif Mohammad
Subject: Re: Cleaning up the PBS/Torque queues
On 13/06/12 17:30, Kashif Mohammad wrote:
> Hi John
>
> It is a longstanding issue with CREAM/PBS and Stephen opened a
> detailed ticket (https://ggus.eu/tech/ticket_show.php?ticket=72506 )
> but it is not fixed yet. At Oxford we regularly kill jobs which
> are either in W state or in Q state but assigned to a WN.
>
> for job in $(qstat | grep " Q " | cut -d. -f1) ; do if ( qstat -f ${job} | grep exec>>/dev/null) ; then qdel -p ${job} ; fi ; done
>
> It will kill any job which is in Q state but assigned to a WN.
>
> One of the issue we have noticed is that some time jobs from lower
> priority VO/users has to stay in queue for long enough to get its
> proxy expired and CREAM doesn't handle this situation properly.
Not more proxy problems :-(.
I know you raised the possibility of this happening at a previous
operations meeting - but I wasn't aware anyone had done tests to
demonstrate this as a problem in addition to the WMS not renewing proxies.
Can I have a GGUS ticket number please.
Thanks,
Chris
>
> Cheers
> Kashif
>
>
> -----Original Message-----
> From: Testbed Support for GridPP member institutes [mailto:[log in to unmask]] On Behalf Of John Hill
> Sent: 13 June 2012 16:37
> To: [log in to unmask]
> Subject: Cleaning up the PBS/Torque queues
>
> While investigating the recent supposed CVMFS and analysis job issues at
> Cambridge, I came across PBS errors in /var/log/messages on the WNs
> which reported copy errors when getting files from the CREAM Sandbox
> area. Further digging has identified these as old pilot jobs (some from
> August last year!) which are still lurking in the PBS queue and are
> being periodically restarted. "showq" indicates that we have about 3500
> of these relic jobs.I was wondering whether there was there a
> recommended way to tidy up the queue?
>
> John
|