What does:
grep -i 598295305 /opt/glite/var/log/glite-ce-cream.log*
report ?
Can you please issue this command on the CREAM CE as user tomcat:
/opt/glite/bin/glite_cream_load_monitor --show
?
Is there a huge number of "Detected value for Number of pending
commands" ?
If so, can you please issue these mysql commands ?
use creamdb;
select c.name, c.creationTime from JOB_MANAGEMENT jm, command c where
jm.commandId =c.id order by c.creationTime limit 20;
select c.name, count(c.name) from JOB_MANAGEMENT jm, command c where
jm.commandId =c.id group by c.name;
Cheers, Massimo
On Wed, 13 Oct 2010, Maarten van Ingen wrote:
> Hi,
>
> One of our creamce keeps jobs in registered state and many will not come out
> of it.
> Sometimes they will get through, but this could take some hours.
>
> For example this job:
> maarten$ glite-ce-job-submit -a -r creamce.gina.sara.nl:8443/cream-pbs-infra
> ./gina
> 2010-10-13 15:47:25,246 WARN - No configuration file suitable for loading. Using
> built-in configuration
> https://creamce.gina.sara.nl:8443/CREAM598295305
>
>
>
> maarten$ glite-ce-job-status https://creamce.gina.sara.nl:8443/CREAM598295305
> 2010-10-13 15:49:12,791 WARN - No configuration file suitable for loading. Using
> built-in configuration
>
> ****** JobID=[https://creamce.gina.sara.nl:8443/CREAM598295305]
> Status = [REGISTERED]
>
>
> When I have a look into the logging, all I can find is this:
> root# grep 598295305 glite-ce-cream.log
> 13 Oct 2010 15:47:27,553 INFO
> org.glite.ce.cream.jobmanagement.db.table.JobTable (JobTable.java:232) -
> (http-8443-Processor19) Job inserted. JobId = CREAM598295305
> 13 Oct 2010 15:47:27,661 INFO
> org.glite.ce.creamapi.jobmanagement.cmdexecutor.AbstractJobExecutor
> (AbstractJobExecutor.java:2094) - (http-8443-Processor19) JOB CREAM598295305
> STATUS CHANGED: -- => REGISTERED [localUser=pvi032]
> [delegationId=ce2ca4874b98dd5f6b55c9e6b3b4a4a1f852d36c]
>
>
> The jdl used is the same as I use to submit to a wms (hence the "Requirements"
> part):
>
> Executable = "/bin/env";
> Arguments = "| /bin/mail -s $(hostname) [log in to unmask]";
> Stdoutput = "message.txt";
> StdError = "stderror";
> Requirements = other.GlueCEUniqueID == "creamce.gina.sara.nl:8443/cream-pbs-
> infra";
> RetryCount=0;
> ShallowRetryCount=0;
>
>
> Also when I use bogus information for the requested queue it stays in the
> REGISTERED state.:
>
> maarten$ glite-ce-job-submit -a -r creamce.gina.sara.nl:8443/cream-pbs-
> thisisbogus ./gina
> 2010-10-13 15:57:55,017 WARN - No configuration file suitable for loading. Using
> built-in configuration
> https://creamce.gina.sara.nl:8443/CREAM392820764
>
> maarten$ glite-ce-job-status https://creamce.gina.sara.nl:8443/CREAM392820764
> 2010-10-13 15:58:08,130 WARN - No configuration file suitable for loading. Using
> built-in configuration
>
> ****** JobID=[https://creamce.gina.sara.nl:8443/CREAM392820764]
> Status = [REGISTERED]
>
>
> Anyone got an idea on whats going on?
> I have the feeling this is something small I am overlooking :-) but it keeps
> me busy.
>
> Cheers,
> Maarten
>
> --
> ing. M.H. van Ingen, HPC&V Systems Programmer
>
> SARA Computing and Networking Services
> PO Box 94613
> 1090 GP Amsterdam, Netherlands
>
> Tel: +31 (0)20 592 3000
> Fax: +31 (0)20 668 3167
>
\|||/
-----------0oo----( o o )----oo0-------------------
(_)
INFN Sezione di Padova
Via Marzolo, 8
35131 Padova - Italy E-mail: massimo.sgaravatto [at] pd.infn.it
Tel: ++39 0498275908 Skype: massimo.sgaravatto
Fax: ++39 0498275952
|