We'll give them a try.
Thanks.
Antonio.
Gergely Debreczeni escribió:
> Hi !
>
> May these 2 patch and the related bugs can explain (at least a part
> of) your problem:
>
> https://savannah.cern.ch/patch/?821
> https://savannah.cern.ch/patch/?897
>
> Gergo
>
> Antonio Delgado Peris wrote:
>> Dear all,
>>
>> For quite a long time, we (CIEMAT-LCG2) experience the known
>> tomcat/rgma memory problems that make it necessary to restart tomcat
>> every week or so. However, since yesterday such problems have become
>> much more severe, and now our tomcat (in lcg03.ciemat.es) dies every
>> few hours. We don't know what may be the cause for this, since we are
>> not aware of any significant change lately.
>>
>> We see in the logs several times:
>> SEVERE: An exception or error occurred in the container during the
>> request processing java.lang.OutOfMemoryError
>>
>> And just before dying:
>> An unexpected exception has been detected in native code outside
>> the VM.
>> Unexpected Signal : 11 occurred at PC=0xB75C60C9
>>
>> In the tomcat and R-GMA logs, we see errors like:
>> java.lang.NumberFormatException: For input string: ""
>> java.lang.NumberFormatException: multiple points
>> java.lang.NumberFormatException: For input string: "211006E4.211006E4"
>> But they are probably not related.
>>
>> Some hints: after every restart, the number of tomcat threads (for a
>> single process) grows very fast, and with it the memory consumption.
>> Also we see an ever increasing number of connections to
>> rgma12.pp.rl.ac.uk. With netstat -n:
>> tcp 0 0 130.206.11.205:34556
>> 130.246.43.212:8088 ESTABLISHED
>> E.g.: on a given moment, the number of threads is 863, and the number
>> of connections is 820.
>> We didn't check this when tomcat died slowly (in a week), so we can't
>> compare the values.
>>
>> We have more jobs running in our farm lately and we see that the WNs
>> contact the monbox every once in a while, but surely not so many to
>> cause this problem. We have also tried turning off rgma-gin and the
>> problem remains the same.
>>
>> And, regarding changes, the latest updated packets were:
>> bdii-3.8.5-1_sl3 Tue 24 Oct 2006
>> 12:52:24 PM CEST
>> openldap-servers-2.0.27-22 Tue 24 Oct 2006 12:52:13
>> PM CEST
>> lcg-CA-1.10-1 Thu 19 Oct 2006
>> 04:10:59 AM CEST
>> glite-version-3.0.2-1 Tue 12 Sep 2006
>> 10:12:05 AM CEST
>> glite-yaim-3.0.0-22 Tue 12 Sep 2006
>> 10:12:01 AM CEST
>> lcg-version-3.0.2-1 Tue 12 Sep 2006
>> 10:12:00 AM CEST
>> glite-rgma-gin-5.0.7-1 Tue 12 Sep 2006
>> 10:11:49 AM CEST
>> glite-rgma-api-c-5.0.8-1 Tue 12 Sep 2006
>> 10:11:48 AM CEST
>> glite-rgma-standard-tables-5.0.4-1 Tue 12 Sep 2006 10:11:47 AM CEST
>> glite-rgma-server-servlet-5.0.31-1 Tue 12 Sep 2006 10:11:42 AM
>> CEST
>> But, again, it's strange that changes applied on 12th september
>> started to show up only yesterday.
>>
>> Ah, our java version is 1.4.2_08
>>
>> Any ideas?
>>
>> Thanks in advance.
>>
>> Antonio.
|