On Mon, 14 Feb 2005, Frederic Schaer wrote:
> As far as I know, no : it's not a known issue *yet* ;)
:)
> What's the output of /opt/edg/etc/cron/edg-rgma-servlet-monitor (if any...) ?
No output:
[root@skurut18 root]# /opt/edg/etc/cron/edg-rgma-servlet-monitor
[root@skurut18 root]#
> Is there something wrong in /opt/edg/var/log/edg-rgma-tools.log ?
I am attaching the file to this mail. There are some quite suspicious
messages, but not being too familiar with R-GMA, I am not able to deduce
too much from it.
The timestamps are from the time when I was installing and configuring the
R-GMAS machinery, there don't seem to be any messages from later time.
> Could you "echo" the "CMD" run by /opt/edg/etc/cron/edg-rgma-servlet-monitor
> (line 69 for me) to be sure the servlet status publishing script is called
> with the right parameters (why isn't there any (damn) debug option in all
> those Grid scripts ???) ?
Theer are two $CMD calls in the script - as far as I can see, the one on
line 69 doesn't even get called in my situation.
I am including the output of running the scirpt with #!/bin/bash -x to
help to diagnose what's wrong:
+ . /etc/sysconfig/edg
++ EDG_LOCATION=/opt/edg
++ EDG_LOCATION_VAR=/opt/edg/var
++ EDG_TMP=/tmp
++ X509_USER_CERT=/etc/grid-security/hostcert.pem
++ X509_USER_KEY=/etc/grid-security/hostkey.pem
++ GRIDMAP=/etc/grid-security/grid-mapfile
++ GRIDMAPDIR=/etc/grid-security/gridmapdir/
+ logfile=/opt/edg/var/log/edg-rgma-tools.log
+ . /opt/edg/var/edg-rgma/rgma-defaults
++ RGMA_PROPS=/opt/edg/var/edg-rgma
++ CATALINA_HOME=/var/tomcat4
++ JUNIT_JAR=_undefined_
++ LOG4J_JAR=/usr/share/java/log4j.jar
++ MYSQL_JAR=/usr/share/java/mm.mysql-2.0.14.jar
++ NETLOGGER_JAR=/usr/share/java/netlogger.jar
++ PREVAYLER_JAR=/usr/share/java/prevayler.jar
++
EDG_TRUST_SERVER_JAR=/opt/edg/share/java/edg-java-security-trustmanager.jar
++
EDG_AUTHORIZ_SERVER_JAR=/opt/edg/share/java/edg-java-security-authorization.jar
++
EDG_TRUST_CLIENT_JAR=/opt/edg/share/java/edg-java-security-trustmanager-client.jar
++ BOUNCYCASTLE_JAR=/usr/share/java/bcprov-jdk14.jar
++ JAVACC_HOME=_undefined_
++ TEXDOCLET_JAR=_undefined_
++ INFO_JAR=/opt/edg/share/java/info.jar
++ SQLUTIL_JAR=/opt/edg/share/java/sqlutil.jar
++ rgma_c_API=_undefined_
++ RGMA_RPM_VERSION=3.4.35
++ GCC_VERSION=
++ EDG_LOCATION=/opt/edg
++ EDG_LOCATION_VAR=/opt/edg/var
++ SERLOC=skurut18.cesnet.cz
++ REGLOC=lcgic01.gridpp.rl.ac.uk
++ SECURE=n
++ RegistryHosts=lcgic01.gridpp.rl.ac.uk
++ RGMA_ROOT=/opt
+ . /opt/edg/var/edg-rgma/rgma-tools-defaults
++ SS_UPDATE_PERIOD=60
++ VOS=EDG
++ NETMON=localhost.localdomain
++ GRID_INFO_USER=edginfo
++ GOUT_JAR=/opt/edg/share/java/Gout.jar
++ GIN_JAR=/opt/edg/share/java/Gin.jar
++ SDHA_JAR=/opt/edg/share/java/ServiceDataHistoryArchiver.jar
++ SITE_INFO=yes
++ CE_PRESENT=yes
++ SE_PRESENT=yes
++ SITE_NAME=changeme.invalid
++ [log in to unmask]
+ SECURE_OPTION=n
+ ENV_SETUP=/opt/edg/etc/profile.d/edg-rgma-env.sh
+ '[' n = y ']'
+ TMP_DIR=/opt/edg/var/edg-rgma/tmp
+ '[' -d /opt/edg/var/edg-rgma/tmp/ ']'
+ '[' -f /var/tomcat4/webapps/R-GMA/WEB-INF/web.xml ']'
+ '[' -f /opt/edg/var/edg-rgma/tmp/web.xml ']'
+ rm -f /opt/edg/var/edg-rgma/tmp/web.xml
+ cp /var/tomcat4/webapps/R-GMA/WEB-INF/web.xml
/opt/edg/var/edg-rgma/tmp/web.xml
+ chown edginfo /opt/edg/var/edg-rgma/tmp/web.xml
+ CMD=. /opt/edg/etc/profile.d/edg-rgma-env.sh &&
/opt/edg/libexec/edg-rgma-servlet-status -ca
/opt/edg/var/edg-rgma/tmp/web.xml -secure n > /dev/null 2>>
/opt/edg/var/log/edg-rgma-tools.log
+ su - edginfo -c '. /opt/edg/etc/profile.d/edg-rgma-env.sh &&
/opt/edg/libexec/edg-rgma-servlet-status -ca
/opt/edg/var/edg-rgma/tmp/web.xml -secure n > /dev/null 2>>
/opt/edg/var/log/edg-rgma-tools.log'
+ '[' 0 -gt 0 ']'
++ find /opt/edg/var/edg-rgma/tmp/ -mmin +60 -maxdepth 1 -name
servlets-service.time
++ wc -l
+ n= 0
+ '[' 0 -eq 1 -o '!' -f /opt/edg/var/edg-rgma/tmp/servlets-service.time
']'
> Thanks
> Frederic
Thanks a lot for your help in advance.
> P.S : I CC JRA1 so that someone could maybe help us find what's wrong...
Unfortunately lcg-rollout mailinglist strips off the CC:, so I couldn't
send this mail as proper followup to them ...
--
Jiri Kosina
Institute of Physics, Academy of sciences of the Czech Republic
Sat Feb 12 22:00:45 UTC 2005: Starting Service Status
Sat Feb 12 22:00:45 UTC 2005: Service Status Started OK
Mon Feb 14 13:25:34 UTC 2005: ServiceStatusPublisher is aborting because of a fatal error
Mon Feb 14 13:25:34 UTC 2005: ServiceStatusPublisher - Servlet not accessible, API has been closed
org.edg.info.RGMAException: Servlet not accessible, API has been closed
at org.edg.info.APIBase.sendCommand(APIBase.java:427)
at org.edg.info.APIBase.sendCommand(APIBase.java:371)
at org.edg.info.APIBase.showSignOfLife(APIBase.java:261)
at org.edg.info.service.ServiceStatusPublisher.run(ServiceStatusPublisher.java:314)
at org.edg.info.service.ServiceStatusPublisher.main(ServiceStatusPublisher.java:127)
Caused by: org.edg.info.RGMAException: Object has been closed: 432756506
at org.edg.info.XMLSAXConverter.endElement(XMLSAXConverter.java:251)
at org.apache.crimson.parser.Parser2.maybeElement(Parser2.java:1712)
at org.apache.crimson.parser.Parser2.content(Parser2.java:1963)
at org.apache.crimson.parser.Parser2.maybeElement(Parser2.java:1691)
at org.apache.crimson.parser.Parser2.parseInternal(Parser2.java:667)
at org.apache.crimson.parser.Parser2.parse(Parser2.java:337)
at org.apache.crimson.parser.XMLReaderImpl.parse(XMLReaderImpl.java:448)
at org.edg.info.XMLSAXConverter.convertXMLResponse(XMLSAXConverter.java:175)
at org.edg.info.APIBase.sendCommand(APIBase.java:419)
... 4 more
Mon Feb 14 13:31:10 UTC 2005: ServiceStatusPublisher dead but subsys locked
Mon Feb 14 13:31:13 UTC 2005: Service Status Stopping
Mon Feb 14 13:31:16 UTC 2005: Failed to Stop Service Status
Mon Feb 14 13:31:17 UTC 2005: Starting Service Status
Mon Feb 14 13:31:18 UTC 2005: Service Status Started OK
Mon Feb 14 13:47:06 UTC 2005: ServiceStatusPublisher dead but subsys locked
Mon Feb 14 13:47:10 UTC 2005: Service Status Stopping
Mon Feb 14 13:47:12 UTC 2005: Failed to Stop Service Status
Mon Feb 14 13:47:14 UTC 2005: Starting Service Status
Mon Feb 14 13:47:14 UTC 2005: Service Status Started OK
Mon Feb 14 14:14:51 UTC 2005: ServiceStatusPublisher is aborting because of a fatal error
Mon Feb 14 14:14:51 UTC 2005: ServiceStatusPublisher - Servlet not accessible, API has been closed
org.edg.info.RGMAException: Servlet not accessible, API has been closed
at org.edg.info.APIBase.sendCommand(APIBase.java:427)
at org.edg.info.APIBase.sendCommand(APIBase.java:371)
at org.edg.info.APIBase.showSignOfLife(APIBase.java:261)
at org.edg.info.service.ServiceStatusPublisher.run(ServiceStatusPublisher.java:314)
at org.edg.info.service.ServiceStatusPublisher.main(ServiceStatusPublisher.java:127)
Caused by: org.edg.info.RGMAException: Object has been closed: 432756531
at org.edg.info.XMLSAXConverter.endElement(XMLSAXConverter.java:251)
at org.apache.crimson.parser.Parser2.maybeElement(Parser2.java:1712)
at org.apache.crimson.parser.Parser2.content(Parser2.java:1963)
at org.apache.crimson.parser.Parser2.maybeElement(Parser2.java:1691)
at org.apache.crimson.parser.Parser2.parseInternal(Parser2.java:667)
at org.apache.crimson.parser.Parser2.parse(Parser2.java:337)
at org.apache.crimson.parser.XMLReaderImpl.parse(XMLReaderImpl.java:448)
at org.edg.info.XMLSAXConverter.convertXMLResponse(XMLSAXConverter.java:175)
at org.edg.info.APIBase.sendCommand(APIBase.java:419)
... 4 more
Mon Feb 14 14:16:52 UTC 2005: ServiceStatusPublisher dead but subsys locked
Mon Feb 14 14:16:54 UTC 2005: Service Status Stopping
Mon Feb 14 14:16:57 UTC 2005: Failed to Stop Service Status
Mon Feb 14 14:16:59 UTC 2005: Starting Service Status
Mon Feb 14 14:17:00 UTC 2005: Service Status Started OK
Mon Feb 14 15:36:06 UTC 2005: edg-rgma-service-status-monitor restarting Publog
Mon Feb 14 15:36:06 UTC 2005: Restarting Publog
Mon Feb 14 15:36:06 UTC 2005: Publog Stopping
Mon Feb 14 15:36:09 UTC 2005: Publog Stopped OK
Mon Feb 14 15:36:10 UTC 2005: Starting Publog
Mon Feb 14 15:36:10 UTC 2005: Publog Started OK
Mon Feb 14 15:45:56 UTC 2005: ServiceStatusPublisher is aborting because of a fatal error
Mon Feb 14 15:45:56 UTC 2005: ServiceStatusPublisher - Servlet not accessible, API has been closed
org.edg.info.RGMAException: Servlet not accessible, API has been closed
at org.edg.info.APIBase.sendCommand(APIBase.java:427)
at org.edg.info.APIBase.sendCommand(APIBase.java:371)
at org.edg.info.APIBase.showSignOfLife(APIBase.java:261)
at org.edg.info.service.ServiceStatusPublisher.run(ServiceStatusPublisher.java:314)
at org.edg.info.service.ServiceStatusPublisher.main(ServiceStatusPublisher.java:127)
Caused by: org.edg.info.RGMAException: Object has been closed: 432756532
at org.edg.info.XMLSAXConverter.endElement(XMLSAXConverter.java:251)
at org.apache.crimson.parser.Parser2.maybeElement(Parser2.java:1712)
at org.apache.crimson.parser.Parser2.content(Parser2.java:1963)
at org.apache.crimson.parser.Parser2.maybeElement(Parser2.java:1691)
at org.apache.crimson.parser.Parser2.parseInternal(Parser2.java:667)
at org.apache.crimson.parser.Parser2.parse(Parser2.java:337)
at org.apache.crimson.parser.XMLReaderImpl.parse(XMLReaderImpl.java:448)
at org.edg.info.XMLSAXConverter.convertXMLResponse(XMLSAXConverter.java:175)
at org.edg.info.APIBase.sendCommand(APIBase.java:419)
... 4 more
Mon Feb 14 15:46:12 UTC 2005: Publog is aborting because of a fatal error
Could not insert records
org.edg.info.RGMAException: Servlet not accessible, API has been closed
org.edg.info.RGMAException: Servlet not accessible, API has been closed
at org.edg.info.APIBase.sendCommand(APIBase.java:427)
at org.edg.info.Insertable.insert(Insertable.java:189)
at org.edg.info.Insertable.insert(Insertable.java:152)
at org.edg.info.publog.Producer.insert(Producer.java:111)
at org.edg.info.publog.Publog.start(Publog.java:114)
at org.edg.info.publog.Publog.main(Publog.java:43)
Caused by: org.edg.info.RGMAException: Object has been closed: 432756533
at org.edg.info.XMLSAXConverter.endElement(XMLSAXConverter.java:251)
at org.apache.crimson.parser.Parser2.maybeElement(Parser2.java:1712)
at org.apache.crimson.parser.Parser2.content(Parser2.java:1963)
at org.apache.crimson.parser.Parser2.maybeElement(Parser2.java:1691)
at org.apache.crimson.parser.Parser2.parseInternal(Parser2.java:667)
at org.apache.crimson.parser.Parser2.parse(Parser2.java:337)
at org.apache.crimson.parser.XMLReaderImpl.parse(XMLReaderImpl.java:448)
at org.edg.info.XMLSAXConverter.convertXMLResponse(XMLSAXConverter.java:175)
at org.edg.info.APIBase.sendCommand(APIBase.java:419)
... 5 more
Mon Feb 14 15:46:12 UTC 2005: Publog is cleaning up its producers before terminating
|
