I am including the R-GMA support list ...
Steve
> -----Original Message-----
> From: LHC Computer Grid - Rollout
> [mailto:[log in to unmask]] On Behalf Of
> Antonio Delgado Peris
> Sent: 16 November 2006 15:19
> To: [log in to unmask]
> Subject: [LCG-ROLLOUT] R-GMA's tomcat dying
>
> Dear all,
>
> For quite a long time, we (CIEMAT-LCG2) experience the known
> tomcat/rgma memory problems that make it necessary to restart
> tomcat every week or so. However, since yesterday such
> problems have become much more severe, and now our tomcat (in
> lcg03.ciemat.es) dies every few hours. We don't know what may
> be the cause for this, since we are not aware of any
> significant change lately.
>
> We see in the logs several times:
> SEVERE: An exception or error occurred in the container
> during the request processing java.lang.OutOfMemoryError
>
> And just before dying:
> An unexpected exception has been detected in native code
> outside the VM.
> Unexpected Signal : 11 occurred at PC=0xB75C60C9
>
> In the tomcat and R-GMA logs, we see errors like:
> java.lang.NumberFormatException: For input string: ""
> java.lang.NumberFormatException: multiple points
> java.lang.NumberFormatException: For input string:
> "211006E4.211006E4"
> But they are probably not related.
>
> Some hints: after every restart, the number of tomcat threads
> (for a single process) grows very fast, and with it the
> memory consumption.
> Also we see an ever increasing number of connections to
> rgma12.pp.rl.ac.uk. With netstat -n:
> tcp 0 0 130.206.11.205:34556
> 130.246.43.212:8088 ESTABLISHED
> E.g.: on a given moment, the number of threads is 863, and
> the number of connections is 820.
> We didn't check this when tomcat died slowly (in a week), so
> we can't compare the values.
>
> We have more jobs running in our farm lately and we see that
> the WNs contact the monbox every once in a while, but surely
> not so many to cause this problem. We have also tried turning
> off rgma-gin and the problem remains the same.
>
> And, regarding changes, the latest updated packets were:
> bdii-3.8.5-1_sl3 Tue 24 Oct 2006
> 12:52:24 PM CEST
> openldap-servers-2.0.27-22 Tue 24 Oct 2006
> 12:52:13 PM
> CEST
> lcg-CA-1.10-1 Thu 19 Oct 2006
> 04:10:59 AM CEST
> glite-version-3.0.2-1 Tue 12 Sep 2006
> 10:12:05 AM CEST
> glite-yaim-3.0.0-22 Tue 12 Sep 2006
> 10:12:01 AM CEST
> lcg-version-3.0.2-1 Tue 12 Sep 2006
> 10:12:00 AM CEST
> glite-rgma-gin-5.0.7-1 Tue 12 Sep
> 2006 10:11:49
> AM CEST
> glite-rgma-api-c-5.0.8-1 Tue 12 Sep
> 2006 10:11:48
> AM CEST
> glite-rgma-standard-tables-5.0.4-1 Tue 12 Sep 2006
> 10:11:47 AM CEST
> glite-rgma-server-servlet-5.0.31-1 Tue 12 Sep 2006
> 10:11:42 AM CEST
> But, again, it's strange that changes applied on 12th
> september started to show up only yesterday.
>
> Ah, our java version is 1.4.2_08
>
> Any ideas?
>
> Thanks in advance.
>
> Antonio.
>
|