Hi,
Ok, I see
15 Dec 2012 11:33:11,203
org.glite.ce.creamapi.jobmanagement.cmdexecutor.JobSubmissionManager -
gliteCreamLoadMonitor: exitCode = 1 messageError = Threshold for FTP
Connection: 30 => Detected value for FTP Connection: 51
I guess this does not show up with manual running of the script
because it quickly falls below threshold after disabling. This begs
the question how long before it would be set online again by cream -
5mins maybe?
15 Dec 2012 11:38:51,506
org.glite.ce.creamapi.jobmanagement.cmdexecutor.JobSubmissionManager -
AcceptNewJobs = false
..
15 Dec 2012 11:43:45,589
org.glite.ce.creamapi.jobmanagement.cmdexecutor.JobSubmissionManager -
AcceptNewJobs by script = true
In the meanwhile CondorG jobs have failed and block submissions for an
hour (I guess Condor does not respect the accept submission flag). So
the only way around is to avoid the initial disabling - how to
increase this threshold?
/etc/glite-ce-cream-utils/glite_cream_load_monitor.cong
Torsten can you try increasing
FTPConn = 30
Cheers,
Rod.
On 15 December 2012 12:00, Massimo Sgaravatto
<[log in to unmask]> wrote:
> Submissions can get disabled because of the glite_cream_load_monitor script,
> or because an admin issued the glite-ce-disable-submission command
>
> In glite-ce-cream.log* files it is reported when and why submissions are
> disabled
>
>
> E.g.:
>
> 29 Nov 2012 02:39:28,045 INFO
> org.glite.ce.creamapi.jobmanagement.cmdexecutor.\
> JobSubmissionManager (JobSubmissionManager.java:131) - (TIMER) AcceptNewJobs
> by\
> script = false
>
>
> and a few lines above there is the reason:
>
> 29 Nov 2012 02:39:27,386 INFO
> org.glite.ce.creamapi.jobmanagement.cmdexecutor.\
> JobSubmissionManager (JobSubmissionManager.java:187) - (TIMER)
> gliteCreamLoadMo\
> nitor: exitCode = 1 messageError = Threshold for FTP Connection: 50 =>
> Detected\
> value for FTP Connection: 72
>
>
> Cheers, Massimo
>
>
>
> On 12/15/2012 11:46 AM, Sean Crosby wrote:
>>
>>
>>
>> On 15 December 2012 21:38, Torsten Harenberg
>> <[log in to unmask]
>> <mailto:[log in to unmask]>> wrote:
>>
>> Hi Sean,
>>
>> thanks for the update.
>>
>> Am 15.12.2012 um 11:27 schrieb Sean Crosby <[log in to unmask]
>> <mailto:[log in to unmask]>>:
>>
>>
>> > Relevant?
>>
>> I fear not, we are using SGE and this gets it's commands and libs
>> from a shared NFS directory directly from the SGE master.
>>
>> The point is: it works and then stops for hours. If you heavily
>> restart tomcat (stop it, wait for a couple of minutes, start it
>> again; repeat until it helps - checking the pilot logs), you can get
>> around this, but this is of course not a long-term solution.
>>
>>
>> Yeah. Erming had the same problem. According to the pilot submission
>> logs, it would be up for a while, then the pilot submission logs would
>> show the "Submissions are disabled" message. Restarting (and in his
>> case, reinstalling the rpms) would make it better for a while...
>>
>> Maybe there is another monitor which checks for failed job submissions,
>> and if there are, disables submissions? That would explain why when
>> Erming fixed the Torque library/client mismatch issue, he hasn't had any
>> problems since?
>>
>> I would help more, but I'm running UMD-1 CREAM at the moment, so don't
>> have a machine to check this on...
>>
>> Cheers,
>> Sean
>>
>>
>> It's a pity that due to the other but, you cannot switch off
>> glite_cream_load_monitor at all :(.
>>
>> Thanks again,
>>
>> Torsten
>>
>> --
>> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
>> <> <>
>> <> Dr. Torsten Harenberg [log in to unmask]
>> <mailto:[log in to unmask]> <>
>>
>> <> Bergische Universitaet <>
>> <> FB C - Physik Tel.: +49 (0)202 439-3521
>> <tel:%2B49%20%280%29202%20439-3521> <>
>>
>> <> Gaussstr. 20 Fax : +49 (0)202 439-2811
>> <tel:%2B49%20%280%29202%20439-2811> <>
>>
>> <> 42097 Wuppertal <>
>> <> <>
>> <><><><><><><>< Of course it runs NetBSD http://www.netbsd.org ><>
>>
>>
>>
>>
>>
>> --
>> Sean Crosby
>> Research Computing System Administrator and Developer
>> ARC Centre of Excellence for Particle Physics at the Terascale
>> School of Physics | University of Melbourne Vic 3010
>> T: +61 3 8344 8093
>>
>>
>
>
--
Tel. +49 89 289 14152
|