Dear all,
For quite a long time, we (CIEMAT-LCG2) experience the known tomcat/rgma
memory problems that make it necessary to restart tomcat every week or
so. However, since yesterday such problems have become much more severe,
and now our tomcat (in lcg03.ciemat.es) dies every few hours. We don't
know what may be the cause for this, since we are not aware of any
significant change lately.
We see in the logs several times:
SEVERE: An exception or error occurred in the container during the
request processing java.lang.OutOfMemoryError
And just before dying:
An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0xB75C60C9
In the tomcat and R-GMA logs, we see errors like:
java.lang.NumberFormatException: For input string: ""
java.lang.NumberFormatException: multiple points
java.lang.NumberFormatException: For input string: "211006E4.211006E4"
But they are probably not related.
Some hints: after every restart, the number of tomcat threads (for a
single process) grows very fast, and with it the memory consumption.
Also we see an ever increasing number of connections to
rgma12.pp.rl.ac.uk. With netstat -n:
tcp 0 0 130.206.11.205:34556
130.246.43.212:8088 ESTABLISHED
E.g.: on a given moment, the number of threads is 863, and the number of
connections is 820.
We didn't check this when tomcat died slowly (in a week), so we can't
compare the values.
We have more jobs running in our farm lately and we see that the WNs
contact the monbox every once in a while, but surely not so many to
cause this problem. We have also tried turning off rgma-gin and the
problem remains the same.
And, regarding changes, the latest updated packets were:
bdii-3.8.5-1_sl3 Tue 24 Oct 2006
12:52:24 PM CEST
openldap-servers-2.0.27-22 Tue 24 Oct 2006 12:52:13 PM
CEST
lcg-CA-1.10-1 Thu 19 Oct 2006
04:10:59 AM CEST
glite-version-3.0.2-1 Tue 12 Sep 2006
10:12:05 AM CEST
glite-yaim-3.0.0-22 Tue 12 Sep 2006
10:12:01 AM CEST
lcg-version-3.0.2-1 Tue 12 Sep 2006
10:12:00 AM CEST
glite-rgma-gin-5.0.7-1 Tue 12 Sep 2006 10:11:49
AM CEST
glite-rgma-api-c-5.0.8-1 Tue 12 Sep 2006 10:11:48
AM CEST
glite-rgma-standard-tables-5.0.4-1 Tue 12 Sep 2006 10:11:47 AM CEST
glite-rgma-server-servlet-5.0.31-1 Tue 12 Sep 2006 10:11:42 AM CEST
But, again, it's strange that changes applied on 12th september started
to show up only yesterday.
Ah, our java version is 1.4.2_08
Any ideas?
Thanks in advance.
Antonio.
|