The limiter script is run every 10 minutes.
The relevant documentation (where it is also explained how to change the
default threadsolds) is here:
https://wiki.italiangrid.it/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI2#3_14_Self_limiting_CREAM_behavio
Cheers, Massimo
On 12/15/2012 12:32 PM, Rodney Walker wrote:
> Hi,
> Ok, I see
>
> 15 Dec 2012 11:33:11,203
> org.glite.ce.creamapi.jobmanagement.cmdexecutor.JobSubmissionManager -
> gliteCreamLoadMonitor: exitCode = 1 messageError = Threshold for FTP
> Connection: 30 => Detected value for FTP Connection: 51
>
> I guess this does not show up with manual running of the script
> because it quickly falls below threshold after disabling. This begs
> the question how long before it would be set online again by cream -
> 5mins maybe?
>
> 15 Dec 2012 11:38:51,506
> org.glite.ce.creamapi.jobmanagement.cmdexecutor.JobSubmissionManager -
> AcceptNewJobs = false
> ..
> 15 Dec 2012 11:43:45,589
> org.glite.ce.creamapi.jobmanagement.cmdexecutor.JobSubmissionManager -
> AcceptNewJobs by script = true
>
> In the meanwhile CondorG jobs have failed and block submissions for an
> hour (I guess Condor does not respect the accept submission flag). So
> the only way around is to avoid the initial disabling - how to
> increase this threshold?
> /etc/glite-ce-cream-utils/glite_cream_load_monitor.cong
> Torsten can you try increasing
> FTPConn = 30
>
>
> Cheers,
> Rod.
>
> On 15 December 2012 12:00, Massimo Sgaravatto
> <[log in to unmask]> wrote:
>> Submissions can get disabled because of the glite_cream_load_monitor script,
>> or because an admin issued the glite-ce-disable-submission command
>>
>> In glite-ce-cream.log* files it is reported when and why submissions are
>> disabled
>>
>>
>> E.g.:
>>
>> 29 Nov 2012 02:39:28,045 INFO
>> org.glite.ce.creamapi.jobmanagement.cmdexecutor.\
>> JobSubmissionManager (JobSubmissionManager.java:131) - (TIMER) AcceptNewJobs
>> by\
>> script = false
>>
>>
>> and a few lines above there is the reason:
>>
>> 29 Nov 2012 02:39:27,386 INFO
>> org.glite.ce.creamapi.jobmanagement.cmdexecutor.\
>> JobSubmissionManager (JobSubmissionManager.java:187) - (TIMER)
>> gliteCreamLoadMo\
>> nitor: exitCode = 1 messageError = Threshold for FTP Connection: 50 =>
>> Detected\
>> value for FTP Connection: 72
>>
>>
>> Cheers, Massimo
>>
>>
>>
>> On 12/15/2012 11:46 AM, Sean Crosby wrote:
>>>
>>>
>>>
>>> On 15 December 2012 21:38, Torsten Harenberg
>>> <[log in to unmask]
>>> <mailto:[log in to unmask]>> wrote:
>>>
>>> Hi Sean,
>>>
>>> thanks for the update.
>>>
>>> Am 15.12.2012 um 11:27 schrieb Sean Crosby <[log in to unmask]
>>> <mailto:[log in to unmask]>>:
>>>
>>>
>>> > Relevant?
>>>
>>> I fear not, we are using SGE and this gets it's commands and libs
>>> from a shared NFS directory directly from the SGE master.
>>>
>>> The point is: it works and then stops for hours. If you heavily
>>> restart tomcat (stop it, wait for a couple of minutes, start it
>>> again; repeat until it helps - checking the pilot logs), you can get
>>> around this, but this is of course not a long-term solution.
>>>
>>>
>>> Yeah. Erming had the same problem. According to the pilot submission
>>> logs, it would be up for a while, then the pilot submission logs would
>>> show the "Submissions are disabled" message. Restarting (and in his
>>> case, reinstalling the rpms) would make it better for a while...
>>>
>>> Maybe there is another monitor which checks for failed job submissions,
>>> and if there are, disables submissions? That would explain why when
>>> Erming fixed the Torque library/client mismatch issue, he hasn't had any
>>> problems since?
>>>
>>> I would help more, but I'm running UMD-1 CREAM at the moment, so don't
>>> have a machine to check this on...
>>>
>>> Cheers,
>>> Sean
>>>
>>>
>>> It's a pity that due to the other but, you cannot switch off
>>> glite_cream_load_monitor at all :(.
>>>
>>> Thanks again,
>>>
>>> Torsten
>>>
>>> --
>>> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
>>> <> <>
>>> <> Dr. Torsten Harenberg [log in to unmask]
>>> <mailto:[log in to unmask]> <>
>>>
>>> <> Bergische Universitaet <>
>>> <> FB C - Physik Tel.: +49 (0)202 439-3521
>>> <tel:%2B49%20%280%29202%20439-3521> <>
>>>
>>> <> Gaussstr. 20 Fax : +49 (0)202 439-2811
>>> <tel:%2B49%20%280%29202%20439-2811> <>
>>>
>>> <> 42097 Wuppertal <>
>>> <> <>
>>> <><><><><><><>< Of course it runs NetBSD http://www.netbsd.org ><>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sean Crosby
>>> Research Computing System Administrator and Developer
>>> ARC Centre of Excellence for Particle Physics at the Terascale
>>> School of Physics | University of Melbourne Vic 3010
>>> T: +61 3 8344 8093
>>>
>>>
>>
>>
>
>
>
|