Hi All ...
My latest EMI2 CreamCE (Centos6, everythign updated to the latest
EMI2 releases) devoted to CMS gets stuck once a day. The very
simple SAM test to check the certificate fails, meaning that the web
service is not responding or active.
I'm able to log to the machine but I see all the memory consumed and
loads around 20. In catalina.out I also see the following message:
INFO: Maximum number of threads (500)
created for connector with address null and port 8443
I've restarted gLite service, which temporary fixes the issue, but
it reappears after a day. I've also decided to increase the number
of thread
that tomcat can deal with (in /etc/tomcat6/server.xml) but, after a
day, I got once again
INFO: Maximum number of threads (500)
created for connector with address null and port 8443
Independently of being in a stuck situation or not, a restart of
gLite service gives me a lot of SEVERE warnings in catalina.out,
like the ones
bellow:
SEVERE: A web application appears to have started a thread
named [Timer-4] but has failed to stop it. This is very likely
to create a memory leak.
Jun 3, 2013 4:08:52 PM
org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
or
SEVERE: A web application created a ThreadLocal with key of
type [null] (value
[org.apache.axiom.util.UIDGenerator$1@6e03923d]) and a value of
type [long[]] (value [[J@2c761b38]) but failed to remove it when
the web application was stopped. To prevent a memory leak, the
ThreadLocal has been forcibly removed.
Jun 3, 2013 4:08:52 PM
org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with
key of type
[org.glite.ce.cream.jobmanagement.cmdexecutor.blah.BLAHExecutor.MyThreadLocal]
(value
[org.glite.ce.cream.jobmanagement.cmdexecutor.blah.BLAHExecutor$MyThreadLocal@7df51055])
and a value of type [java.lang.UNIXProcess] (value
[java.lang.UNIXProcess@64037d0d]) but failed to remove it when
the web application was stopped. To prevent a memory leak, the
ThreadLocal has been forcibly removed.
Something is not right in the thread management within tomcat. It
seems thread are not properly released and keep increasing
until all the memory of the machine is consumed.
I've already opened a ticket in GGUS
https://ggus.eu/tech/ticket_show.php?ticket=94517
but I'm unsure about the level of support CREAM PT is now providing,
and therefore, decided also to put the issue here.
Help is appreciated to avoid restarting service once a day.
Cheers
Goncalo
