Hello,
I see again in catalina.out error messages
....
Stopping service Tomcat-Insecure
Starting service Tomcat-Insecure
Apache Tomcat/4.1
log4j:WARN No appenders could be found for logger
(org.edg.info.ArchiverServlet).
log4j:WARN Please initialize the log4j system properly.
java.net.SocketException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
.....
These messages appeared today Mar 23 at 00:33.
In file /var/log/boot.log I see
....
Мар 23 00:32:32 pcfarm last message repeated 2 times
Мар 23 00:33:34 pcfarm su: Using CATALINA_BASE: /var/tomcat4
Мар 23 00:33:34 pcfarm su: Using CATALINA_HOME: /var/tomcat4
Мар 23 00:33:34 pcfarm su: Using CATALINA_TMPDIR: /var/tomcat4/temp
Мар 23 00:33:34 pcfarm su: Using JAVA_HOME: /usr/java/j2sdk1.4.2_04
Мар 23 00:33:35 pcfarm tomcat4: startup dtomcat4 succeeded
Мар 23 00:33:52 pcfarm su: Using CATALINA_BASE: /var/tomcat4
Мар 23 00:33:52 pcfarm su: Using CATALINA_HOME: /var/tomcat4
Мар 23 00:33:52 pcfarm su: Using CATALINA_TMPDIR: /var/tomcat4/temp
Мар 23 00:33:52 pcfarm su: Using JAVA_HOME: /usr/java/j2sdk1.4.2_04
Мар 23 00:33:52 pcfarm tomcat4: startup dtomcat4 succeeded
Мар 23 00:34:02 pcfarm lcg-bdii: succeeded
Мар 23 00:36:03 pcfarm last message repeated 3 times
Мар 23 00:38:03 pcfarm last message repeated 3 times
Мар 23 00:40:03 pcfarm last message repeated 3 times
Мар 23 00:40:03 pcfarm last message repeated 2 times
Мар 23 00:40:05 pcfarm edg-rgma-publog:
/opt/edg/etc/init.d/edg-rgma-publog shutdown succeeded
Мар 23 00:40:05 pcfarm edg-rgma-publog:
/opt/edg/etc/init.d/edg-rgma-publog startup succeeded
Мар 23 00:42:03 pcfarm lcg-bdii: succeeded
....
Мар 23 00:40:05 pcfarm edg-rgma-publog:
/opt/edg/etc/init.d/edg-rgma-publog shutdown succeeded
Мар 23 00:40:05 pcfarm edg-rgma-publog:
/opt/edg/etc/init.d/edg-rgma-publog startup succeeded
Мар 23 00:42:03 pcfarm lcg-bdii: succeeded
Command ps -ef|grep tomcat4 gives:
tomcat4 19265 1 0 Mar22 pts/1 00:00:01 /usr/java/j2sdk1.4.2_04/bin/java
tomcat4 19268 19265 0 Mar22 pts/1 00:00:01 /usr/java/j2sdk1.4.2_04/bin/java
....
tomcat4 9006 1 0 00:33 ? 00:00:01 /usr/java/j2sdk1.4.2_04/bin/java
tomcat4 9012 9006 0 00:33 ? 00:00:01 /usr/java/j2sdk1.4.2_04/bin/java
....
tomcat4 14005 9012 0 00:40 ? 00:00:00 /usr/java/j2sdk1.4.2_04/bin/java
tomcat4 17000 9012 0 00:46 ? 00:00:00 /usr/java/j2sdk1.4.2_04/bin/java
tomcat4 17001 9012 0 00:46 ? 00:00:22 /usr/java/j2sdk1.4.2_04/bin/java
tomcat4 17003 9012 0 00:46 ? 00:00:00 /usr/java/j2sdk1.4.2_04/bin/jav
It looks like today at 00:33 new java proccesses were started but older
ones were not killed and because of that I see error messages in
catalina.out.
My crontab -l shows
*/2 * * * * /opt/edg/etc/cron/edg-rgma-service-status-monitor >>
/opt/edg/var/log/edg-rgma-tools.log 2>&1
*/2 * * * * /opt/edg/etc/cron/edg-rgma-servlet-monitor >>
/opt/edg/var/log/edg-rgma-tools.log 2>&1
*/2 * * * * /opt/lcg/libexec/lcg-bdii-update
/opt/lcg/var/bdii/lcg-bdii.conf > /dev/null 2>&1
File /opt/edg/var/log/edg-rgma-tools.log is attached to the mail.
Date in this file is given in UTC but my local time UTC+3.
I have an impression that such situation happens when coming to next
day.
What script did restart catalina and why ?
Why were not killed old proccesses ?
Any hints ?
Thank you.
On Mon, 21 Mar 2005, Anatoly Oreshkin wrote:
> Date: Mon, 21 Mar 2005 20:25:55 +0300
> From: Anatoly Oreshkin <[log in to unmask]>
> Reply-To: LHC Computer Grid - Rollout <[log in to unmask]>
> To: [log in to unmask]
> Subject: Re: [LCG-ROLLOUT] Tomcat4 error messages
>
> Yes, tomcat4 script is on, but it is started when rebooting machine.
> Last time I've run
>
> /etc/rc.d/init.d/edg-tomcat4 stop
> su tomcat4 -c 'killall -9 java'
> rm -f /var/tomcat4/logs/catalina.out
> /etc/rc.d/init.d/edg-tomcat4 start
>
> at the beginning of March however
> uptime shows
> 8:13pm up 27 days, 5:49, 33 users, load average: 0.01, 0.06, 0.08
>
> So should I switch tomcat4 script off ?
> Nevertheless I don't understand the cause.
>
>
>
>
>
> On Mon, 21 Mar 2005, Laurence wrote:
>
> > Date: Mon, 21 Mar 2005 17:38:13 +0100
> > From: Laurence <[log in to unmask]>
> > To: Anatoly Oreshkin <[log in to unmask]>
> > Cc: [log in to unmask]
> > Subject: Re: [LCG-ROLLOUT] Tomcat4 error messages
> >
> > Don't forget that there are two Tomcat startup scripts, tomcat4 and
> > edg-tomcat4
> > I notice in your list that tomcat4 is on but the cron job starts
> > edg-tomcat4.
>
> >
> > They are fighting over the same port.
> >
> >
> >
> > Anatoly Oreshkin wrote:
> >
> > >
> > > Hello,
> > >
> > > I've run commands you proposed and that fixed problem for a while.
> > >
> > > But later on I can't say exactly when the same problem would have
> > > repeated. That is
> > > java.net.SocketException: Address already in use
> > > and so on.
> > >
> > > It looks like tomcat has been started once more without stopping or
> > > tomcat stopping has failed ?
> > >
> > > My crontab is
> > >
> > > */2 * * * * /opt/edg/etc/cron/edg-rgma-service-status-monitor >>
> > > /opt/edg/var/log/edg-rgma-tools.log 2>&1
> > >
> > > */2 * * * * /opt/edg/etc/cron/edg-rgma-servlet-monitor >>
> > > /opt/edg/var/log/edg-rgma-tools.log 2>&1
> > >
> > >
> > > chkconfig --list shows
> > >
> > > globus-gridftp 0:off 1:off 2:off 3:on 4:on 5:on 6:off
> > > rfiod 0:off 1:off 2:off 3:on 4:on 5:on 6:off
> > > globus-gatekeeper 0:off 1:off 2:off 3:on 4:on
> > > 5:on 6:off
> > > globus-mds 0:off 1:off 2:off 3:on 4:on 5:on 6:off
> > > edg-fmon-agent 0:off 1:off 2:off 3:on 4:on 5:on 6:off
> > > edg-wl-locallogger 0:off 1:off 2:off 3:on 4:on
> > > 5:on 6:off
> > > edg-netconf 0:off 1:off 2:off 3:on 4:on 5:on 6:off
> > > pbs_mom 0:off 1:off 2:off 3:on 4:on 5:on 6:off
> > > pbs_server 0:off 1:off 2:off 3:on 4:on 5:on 6:off
> > > edg-rgma-gin 0:off 1:off 2:off 3:off 4:off 5:off 6:off
> > > lcg-bdii 0:off 1:off 2:off 3:on 4:on 5:on 6:off
> > > mysql 0:off 1:off 2:off 3:on 4:on 5:on 6:off
> > > tomcat4 0:off 1:off 2:off 3:on 4:on 5:on 6:off
> > > edg-rgma-service-status 0:off 1:off 2:off 3:off 4:off
> > > 5:off 6:off
> > >
> > > I've installed UI, CE, SE, MON, WN on one machine.
> > >
> > > What might be the cause ?
> > >
> > > Thank you.
> > >
> > > On Wed, 2 Mar 2005, Laurence wrote:
> > >
> > > > Date: Wed, 2 Mar 2005 12:30:56 +0100
> > > > From: Laurence <[log in to unmask]>
> > > > Reply-To: LHC Computer Grid - Rollout <[log in to unmask]>
> > > > To: [log in to unmask]
> > > > Subject: Re: [LCG-ROLLOUT] Tomcat4 error messages
> > > >
> > > > This error means that there is already a process that has binded to
> > > port
> > > > 8080.
> > > > Running the following commands should fix the problem.
> > > >
> > > > /etc/rc.d/init.d/edg-tomcat4 stop
> > > > su tomcat4 -c 'killall -9 java'
> > > > rm -f /var/tomcat4/logs/catalina.out
> > > > /etc/rc.d/init.d/edg-tomcat4 start
> > > >
> > > >
> > > >
> > > > Anatoly Oreshkin wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I've installed on one RH 7.3 machine LCG-2_3_0 UI, CE-torque,
> > > WN-torque,
> > > > > MON, SECLASSIC using aimk scripts.
> > > > > Examining Tomcat4 log catalina.out I've found out warning and error
> > > > > lines:
> > > > >
> > > > > Starting service Tomcat-Insecure
> > > > > Apache Tomcat/4.1
> > > > > log4j:WARN No appenders could be found for logger
> > > > > (org.edg.info.ArchiverServlet).
> > > > > log4j:WARN Please initialize the log4j system properly.
> > > > > java.net.SocketException: Address already in use
> > > > > ...
> > > > >
> > > > > Full log is attached to mail.
> > > > >
> > > > >
> > > > > What might be the cause ?
> > > > > Any hints.
> > > > >
> > > > > Thank you.
> > > > > Anatoly.
> > > > >
> > > > >
> > > > >
> > > >
> > > >------------------------------------------------------------------------
> > > > >
> > > > >Starting service Tomcat-Insecure
> > > > >Apache Tomcat/4.1
> > > > >log4j:WARN No appenders could be found for logger
> > > > >(org.edg.info.ArchiverServlet).
> > > > >log4j:WARN Please initialize the log4j system properly.
> > > > >java.net.SocketException: Address already in use
> > > > > at sun.nio.ch.Net.bind(Native Method)
> > > > > at
> > > >
> > > >sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:108)
> > >
> > > > > at
> > > > >sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> > > > > at
> > > > >sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
> > > > > at
> > > > >org.edg.info.StreamingServer.setupSocket(StreamingServer.java:628)
> > > > > at
> > > org.edg.info.StreamingServer.<init>(StreamingServer.java:102)
> > > > > at org.edg.info.ConsumerServlet.init(ConsumerServlet.java:101)
> > > > > at javax.servlet.GenericServlet.init(GenericServlet.java)
> > > > > at
> > > >
> > > >org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:934)
> > >
> > > > > at
> > > >
> > > >org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:821)
> > > > > at
> > > >
> > > >org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:3420)
> > >
> > > > > at
> > > > >org.apache.catalina.core.StandardContext.start(StandardContext.java:
> > > > >3608)
> > > > > at
> > > >
> > > >org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:821)
> > >
> > > > > at
> > > >
> > > >org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:807)
> > > > > at
> > > > >org.apache.catalina.core.StandardHost.addChild(StandardHost.java:579)
> > > > > at
> > > >
> > > >org.apache.catalina.core.StandardHostDeployer.install(StandardHostDeployer.java:257)
> > >
> > > > > at
> > > > >org.apache.catalina.core.StandardHost.install(StandardHost.java:772)
> > > > >at
> > > >
> > > >org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:569)
> > >
> > > > > at
> > > > >org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:411)
> > > > > at
> > > > >org.apache.catalina.startup.HostConfig.start(HostConfig.java:879)
> > > > > at
> > > >
> > > >org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:368)
> > >
> > > > > at
> > > >
> > > >org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:166)
> > >
> > > > > at
> > > > >org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1196)
> > > > > at
> > > > >org.apache.catalina.core.StandardHost.start(StandardHost.java:738)
> > > > > at
> > > > >org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1188)
> > > > > at
> > > > >org.apache.catalina.core.StandardEngine.start(StandardEngine.java:347)
> > > > > at
> > > >
> > > >org.apache.catalina.core.StandardService.start(StandardService.java:497)
> > > > > at
> > > >
> > > >org.apache.catalina.core.StandardServer.start(StandardServer.java:2189)
> > > > > at
> > > org.apache.catalina.startup.Catalina.start(Catalina.java:512)
> > > > > at
> > > org.apache.catalina.startup.Catalina.execute(Catalina.java:400)
> > > > > at
> > > org.apache.catalina.startup.Catalina.process(Catalina.java:180)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > at
> > > >
> > > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > >
> > > > > at
> > > >
> > > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >
> > > > > at java.lang.reflect.Method.invoke(Method.java:324)
> > > > > at
> > > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:203)
> > > > >java.lang.NullPointerException
> > > > > at
> > > >
> > > >org.edg.info.ContinuousConsumerInstance.prepareStartStreamingMessage(ContinuousConsumerInstance.java:158)
> > >
> > > > > at
> > > >
> > > >org.edg.info.ContinuousConsumerInstance$StartQueryThread.run(ContinuousConsumerInstance.java:417)
> > >
> > > > >java.lang.NullPointerException
> > > > > at
> > > >
> > > >org.edg.info.ContinuousConsumerInstance.prepareStartStreamingMessage(ContinuousConsumerInstance.java:158)
> > >
> > > > > at
> > > >
> > > >org.edg.info.ContinuousConsumerInstance$StartQueryThread.run(ContinuousConsumerInstance.java:417)
> > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Tue Mar 22 10:12:33 UTC 2005: edg-rgma-restart-all Completed OK
Втр Мар 22 10:14:01 UTC 2005: ServiceStatusPublisher dead but subsys locked
Tue Mar 22 10:14:01 UTC 2005: edg-rgma-service-status-monitor aborted as rgma was restarted within the last 5 minutes
Втр Мар 22 10:16:00 UTC 2005: ServiceStatusPublisher dead but subsys locked
Tue Mar 22 10:16:00 UTC 2005: edg-rgma-service-status-monitor aborted as rgma was restarted within the last 5 minutes
Втр Мар 22 10:18:00 UTC 2005: ServiceStatusPublisher dead but subsys locked
Tue Mar 22 10:18:00 UTC 2005: edg-rgma-service-status-monitor restarting Service Status
Втр Мар 22 10:18:00 UTC 2005: Restarting Service Status
Втр Мар 22 10:18:01 UTC 2005: Service Status Stopping
Втр Мар 22 10:18:03 UTC 2005: Failed to Stop Service Status
Втр Мар 22 10:18:03 UTC 2005: Starting Service Status
Втр Мар 22 10:18:04 UTC 2005: Service Status Started OK
Tue Mar 22 10:18:04 UTC 2005: edg-rgma-service-status-monitor restarting Publog
Втр Мар 22 10:18:04 UTC 2005: Restarting Publog
Втр Мар 22 10:18:04 UTC 2005: Publog Stopping
Втр Мар 22 10:18:07 UTC 2005: Publog Stopped OK
Втр Мар 22 10:18:07 UTC 2005: Starting Publog
Втр Мар 22 10:18:07 UTC 2005: Publog Started OK
Tue Mar 22 21:33:34 UTC 2005: /opt/edg/libexec/edg-rgma-servlet-status - Unable to create and declare dummy table before 90 second timeout
Tue Mar 22 21:33:34 UTC 2005 : Servlet Monitoring Script restarting Tomcat
Stopping tomcat4: [60G[ [1;32mОК[0;39m ]
Tue Mar 22 21:33:36 UTC 2005: edg-rgma-restart-all Waiting for 15 seconds to allow tomcat threads to die
/opt/edg/libexec/edg-rgma-restart-all: pidof: command not found
Starting tomcat4: [60G[ [1;32mОК[0;39m ]
Tue Mar 22 21:33:52 UTC 2005: edg-rgma-restart-all Waiting 15 seconds for tomcat to start
Tue Mar 22 21:34:03 UTC 2005: /opt/edg/libexec/edg-rgma-servlet-status - Unable to create and ' + 'declare dummy table
Tue Mar 22 21:34:03 UTC 2005 : Servlet Monitoring Script restarting Tomcat
Tue Mar 22 21:34:03 UTC 2005: edg-rgma-restart-all restart aborted as it was done within last 5 minutes
Tue Mar 22 21:34:07 UTC 2005: edg-rgma-restart-all Moving /opt/edg/var/log/edg-rgma-tools.log to /opt/edg/var/log/edg-rgma-tools.log~
Tue Mar 22 21:34:17 UTC 2005: Publog is aborting because of a fatal error
Could not insert records
org.edg.info.RGMAException: Servlet not accessible, API has been closed
org.edg.info.RGMAException: Servlet not accessible, API has been closed
at org.edg.info.APIBase.sendCommand(APIBase.java:427)
at org.edg.info.Insertable.insert(Insertable.java:189)
at org.edg.info.Insertable.insert(Insertable.java:152)
at org.edg.info.publog.Producer.insert(Producer.java:111)
at org.edg.info.publog.Publog.start(Publog.java:114)
at org.edg.info.publog.Publog.main(Publog.java:43)
Caused by: org.edg.info.RGMAException: Object has been closed: 1020451704
at org.edg.info.XMLSAXConverter.endElement(XMLSAXConverter.java:251)
at org.apache.crimson.parser.Parser2.maybeElement(Parser2.java:1712)
at org.apache.crimson.parser.Parser2.content(Parser2.java:1963)
at org.apache.crimson.parser.Parser2.maybeElement(Parser2.java:1691)
at org.apache.crimson.parser.Parser2.parseInternal(Parser2.java:667)
at org.apache.crimson.parser.Parser2.parse(Parser2.java:337)
at org.apache.crimson.parser.XMLReaderImpl.parse(XMLReaderImpl.java:448)
at org.edg.info.XMLSAXConverter.convertXMLResponse(XMLSAXConverter.java:175)
at org.edg.info.APIBase.sendCommand(APIBase.java:419)
... 5 more
Tue Mar 22 21:34:17 UTC 2005: Publog is cleaning up its producers before terminating
|
