The grep: (of course this example does go one step further...) /opt/glite/var/log/glite-ce-cream.log:13 Oct 2010 16:13:51,234 INFO org.glite.ce.creamapi.jobmanagement.cmdexecutor.AbstractJobExecutor (AbstractJobExecutor.java:826) - (Worker Thread 11) REMOTE_REQUEST_ADDRESS=145.100.5.194; USER_DN=/O=dutchgrid/O=users/O=sara/CN=Maarten Hendrik van Ingen; USER_FQAN={ /pvier/Role=NULL/Capability=NULL; /pvier/infra/Role=NULL/Capability=NULL; }; CMD_NAME=JOB_START; CMD_CATEGORY=JOB_MANAGEMENT; CMD_STATUS=PROCESSING; commandName=JOB_START; cmdExecutorName=BLAHExecutor; userId=_O_dutchgrid_O_users_O_sara_CN_Maarten_Hendrik_van_Ingen_pvier_Role_NULL_Capability_NULL; jobId=CREAM598295305; status=PROCESSING; /opt/glite/var/log/glite-ce-cream.log:13 Oct 2010 16:13:51,287 INFO org.glite.ce.creamapi.jobmanagement.cmdexecutor.AbstractJobExecutor (AbstractJobExecutor.java:2094) - (Worker Thread 11) JOB CREAM598295305 STATUS CHANGED: REGISTERED => PENDING [localUser=pvi032] [delegationId=ce2ca4874b98dd5f6b55c9e6b3b4a4a1f852d36c] /opt/glite/var/log/glite-ce-cream.log.1:13 Oct 2010 15:47:27,553 INFO org.glite.ce.cream.jobmanagement.db.table.JobTable (JobTable.java:232) - (http-8443-Processor19) Job inserted. JobId = CREAM598295305 /opt/glite/var/log/glite-ce-cream.log.1:13 Oct 2010 15:47:27,661 INFO org.glite.ce.creamapi.jobmanagement.cmdexecutor.AbstractJobExecutor (AbstractJobExecutor.java:2094) - (http-8443-Processor19) JOB CREAM598295305 STATUS CHANGED: -- => REGISTERED [localUser=pvi032] [delegationId=ce2ca4874b98dd5f6b55c9e6b3b4a4a1f852d36c] /opt/glite/bin/glite_cream_load_monitor --show: Threshold for Load Average(1 min): 40 => Detected value for Load Average(1 min): 1.06 Threshold for Load Average(5 min): 40 => Detected value for Load Average(5 min): 0.97 Threshold for Load Average(15 min): 20 => Detected value for Load Average(15 min): 0.69 Threshold for Memory Usage: 95 => Detected value for Memory Usage: 17.57% Threshold for Swap Usage: 95 => Detected value for Swap Usage: 0.00% Threshold for Free FD: 500 => Detected value for Free FD: 2386973 Threshold for tomcat FD: 800 => Detected value for Tomcat FD: 269 Threshold for FTP Connection: 30 => Detected value for FTP Connection: 1 Threshold for Number of active jobs: -1 => Detected value for Number of active jobs: 5866 Threshold for Number of pending commands: -1 => Detected value for Number of pending commands: 431 Threshold for Disk Usage: 95% => Detected value for Partition / : 35% SQL: mysql> select c.name, c.creationTime from JOB_MANAGEMENT jm, command c where -> jm.commandId =c.id order by c.creationTime limit 20; +----------------+---------------------+ | name | creationTime | +----------------+---------------------+ | SET_JOB_STATUS | 2010-10-13 14:33:02 | | SET_JOB_STATUS | 2010-10-13 14:33:09 | | SET_JOB_STATUS | 2010-10-13 14:33:59 | | SET_JOB_STATUS | 2010-10-13 14:34:00 | | SET_JOB_STATUS | 2010-10-13 14:35:54 | | SET_JOB_STATUS | 2010-10-13 14:37:11 | | SET_JOB_STATUS | 2010-10-13 14:39:57 | | PROXY_RENEW | 2010-10-13 14:40:41 | | SET_JOB_STATUS | 2010-10-13 14:45:10 | | SET_JOB_STATUS | 2010-10-13 14:46:11 | | SET_JOB_STATUS | 2010-10-13 14:46:13 | | SET_JOB_STATUS | 2010-10-13 14:46:13 | | SET_JOB_STATUS | 2010-10-13 14:46:14 | | SET_JOB_STATUS | 2010-10-13 14:46:17 | | SET_JOB_STATUS | 2010-10-13 14:48:11 | | SET_JOB_STATUS | 2010-10-13 14:48:11 | | SET_JOB_STATUS | 2010-10-13 14:49:15 | | SET_JOB_STATUS | 2010-10-13 14:49:16 | | SET_JOB_STATUS | 2010-10-13 14:50:19 | | JOB_START | 2010-10-13 14:53:37 | +----------------+---------------------+ 20 rows in set (0.00 sec) mysql> select c.name, count(c.name) from JOB_MANAGEMENT jm, command c where -> jm.commandId =c.id group by c.name; +---------------------------+---------------+ | name | count(c.name) | +---------------------------+---------------+ | COPY_NEW_PROXY_TO_SANDBOX | 3 | | JOB_PURGE | 61 | | JOB_START | 190 | | PROXY_RENEW | 539 | | SET_JOB_STATUS | 195 | +---------------------------+---------------+ 5 rows in set (0.00 sec) Cheers, Maarten On Wednesday 13 October 2010 16:14:49 Massimo Sgaravatto - INFN Padova wrote: > What does: > > grep -i 598295305 /opt/glite/var/log/glite-ce-cream.log* > > report ? > > Can you please issue this command on the CREAM CE as user tomcat: > > /opt/glite/bin/glite_cream_load_monitor --show > > ? > > Is there a huge number of "Detected value for Number of pending > commands" ? > If so, can you please issue these mysql commands ? > > use creamdb; > select c.name, c.creationTime from JOB_MANAGEMENT jm, command c where > jm.commandId =c.id order by c.creationTime limit 20; > > select c.name, count(c.name) from JOB_MANAGEMENT jm, command c where > jm.commandId =c.id group by c.name; > > Cheers, Massimo > > On Wed, 13 Oct 2010, Maarten van Ingen wrote: > > Hi, > > > > One of our creamce keeps jobs in registered state and many will not come > > out of it. > > Sometimes they will get through, but this could take some hours. > > > > For example this job: > > maarten$ glite-ce-job-submit -a -r > > creamce.gina.sara.nl:8443/cream-pbs-infra ./gina > > 2010-10-13 15:47:25,246 WARN - No configuration file suitable for > > loading. Using built-in configuration > > https://creamce.gina.sara.nl:8443/CREAM598295305 > > > > > > > > maarten$ glite-ce-job-status > > https://creamce.gina.sara.nl:8443/CREAM598295305 2010-10-13 15:49:12,791 > > WARN - No configuration file suitable for loading. Using built-in > > configuration > > > > ****** JobID=[https://creamce.gina.sara.nl:8443/CREAM598295305] > > > > Status = [REGISTERED] > > > > When I have a look into the logging, all I can find is this: > > root# grep 598295305 glite-ce-cream.log > > 13 Oct 2010 15:47:27,553 INFO > > org.glite.ce.cream.jobmanagement.db.table.JobTable (JobTable.java:232) - > > (http-8443-Processor19) Job inserted. JobId = CREAM598295305 > > 13 Oct 2010 15:47:27,661 INFO > > org.glite.ce.creamapi.jobmanagement.cmdexecutor.AbstractJobExecutor > > (AbstractJobExecutor.java:2094) - (http-8443-Processor19) JOB > > CREAM598295305 STATUS CHANGED: -- => REGISTERED [localUser=pvi032] > > [delegationId=ce2ca4874b98dd5f6b55c9e6b3b4a4a1f852d36c] > > > > > > The jdl used is the same as I use to submit to a wms (hence the > > "Requirements" part): > > > > Executable = "/bin/env"; > > Arguments = "| /bin/mail -s $(hostname) [log in to unmask]"; > > Stdoutput = "message.txt"; > > StdError = "stderror"; > > Requirements = other.GlueCEUniqueID == > > "creamce.gina.sara.nl:8443/cream-pbs- infra"; > > RetryCount=0; > > ShallowRetryCount=0; > > > > > > Also when I use bogus information for the requested queue it stays in the > > REGISTERED state.: > > > > maarten$ glite-ce-job-submit -a -r creamce.gina.sara.nl:8443/cream-pbs- > > thisisbogus ./gina > > 2010-10-13 15:57:55,017 WARN - No configuration file suitable for > > loading. Using built-in configuration > > https://creamce.gina.sara.nl:8443/CREAM392820764 > > > > maarten$ glite-ce-job-status > > https://creamce.gina.sara.nl:8443/CREAM392820764 2010-10-13 15:58:08,130 > > WARN - No configuration file suitable for loading. Using built-in > > configuration > > > > ****** JobID=[https://creamce.gina.sara.nl:8443/CREAM392820764] > > > > Status = [REGISTERED] > > > > Anyone got an idea on whats going on? > > I have the feeling this is something small I am overlooking :-) but it > > keeps me busy. > > > > Cheers, > > Maarten > > > > > > SARA Computing and Networking Services > > PO Box 94613 > > 1090 GP Amsterdam, Netherlands > > > > Tel: +31 (0)20 592 3000 > > Fax: +31 (0)20 668 3167 > > \|||/ > -----------0oo----( o o )----oo0------------------- > (_) > INFN Sezione di Padova > Via Marzolo, 8 > 35131 Padova - Italy E-mail: massimo.sgaravatto [at] pd.infn.it > Tel: ++39 0498275908 Skype: massimo.sgaravatto > Fax: ++39 0498275952 -- ing. M.H. van Ingen, HPC&V Systems Programmer SARA Computing and Networking Services PO Box 94613 1090 GP Amsterdam, Netherlands Tel: +31 (0)20 592 3000 Fax: +31 (0)20 668 3167