Mark
Dave Kant tells me that this is the regular daily check made by gstat to
check whether the site has published data to the accounting repository
recently.
The program first checks that the database table "LcgRecords" is
reachable through rgma, then checks the date of the "newest" record.
If the record is older than 2 days, it flags an Apel error.
Looking at the data at our end shows that this is correct: The last time
we received data from Durham is August 22nd 2005.
| 2005-08-01 | 06:39:40 |
| 2005-08-02 | 23:07:13 |
| 2005-08-03 | 23:59:04 |
| 2005-08-04 | 23:59:58 |
| 2005-08-05 | 23:57:56 |
| 2005-08-06 | 23:59:15 |
| 2005-08-07 | 23:58:08 |
| 2005-08-08 | 23:58:31 |
| 2005-08-09 | 23:58:40 |
| 2005-08-10 | 23:59:57 |
| 2005-08-11 | 23:40:59 |
| 2005-08-12 | 04:43:07 |
| 2005-08-13 | 20:38:37 |
| 2005-08-14 | 01:21:06 |
| 2005-08-17 | 23:30:42 |
| 2005-08-18 | 23:59:31 |
| 2005-08-19 | 09:08:30 |
| 2005-08-20 | 20:15:08 |
| 2005-08-21 | 23:16:47 |
| 2005-08-22 | 02:05:40 |
+------------+----------------+
The only way to see what is happening is to look at the Apel cron jobs
on the CE and MON node. Do the logs from these crons report any
problems?
Jeremy
-----Original Message-----
From: Testbed Support for GridPP member institutes
[mailto:[log in to unmask]] On Behalf Of Mark Nelson
Sent: 01 September 2005 11:38
To: [log in to unmask]
Subject: APEL problems
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello
I'm getting the following error -
APEL Test
Checking if LcgRecords latest archiver is accessible:
+ rgma -c 'select count(*) from LcgRecords'
+ grep 'Rows in set'
+ set +x
LcgRecords latest archiver is working.
Checking if LcgRecords tuples are present for the site (max 3 day old):
++ rgma -c 'select count(*) from LcgRecords where
ExecutingCE='\''helmsley.dur.scotgrid.ac.uk'\'' and
MeasurementDate>='\''2005-08-29'\'''
++ grep '^|'
++ cut -f 2 -d ' '
++ tail -n 1
+ count=0
+ set +x
Count: 0
Empty Apel result set - FAILED!
My log file is showing the following -
/bin/sh: line 1: /opt/edg/sbin/edg-rgma-apel: No such file or directory
Thu Sep 1 02:05:00 UTC 2005: apel-pbs-log-parser - Read-in
configuration: [quiet, d, c, e, recursiveEventFiles, g,
recursiveGKFiles] [DBPassword=****, site=Durham,
pbslog=/var/spool/pbs/server_priv/accounting, CE
host=gppce.test.ac.uk, GIIS=helmsley.dur.scotgrid.ac.uk,
extraEventLogFile=[/home/toucan/pbslog00021123.gz,
/home/toucan/pbslog00021100.log], gklog=/var/log,
DBURL=jdbc:mysql://ayton.dur.scotgrid.ac.uk:3306/accoun
ting, DBUsername=accounting, mslog=/var/log, timeZone=UTC]
Thu Sep 1 02:05:00 UTC 2005: apel-pbs-log-parser - ------ Starting the
apel application ------
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - **** Clearing
database tables ****
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Only removing
processed log records
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Number deleted
processed logs from: EventRecords: 0
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Number deleted
processed logs from: GkRecords: 0
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Number deleted
processed logs from: MessageRecords: 0
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Finding CPU spec
values from GIIS server: helmsley.dur.scotgrid.ac.uk
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Cluster:
helmsley.dur.scotgrid.ac.uk, Subcluster: helmsley.dur.scotgrid.ac.uk,
SpecInt2000: 680, SpecFloat2000: 680
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Spec value remains
unchanged, no need to update it
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Spec for catch-all
updated to specInt: 680 specFloat: 680
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Number of entries
found for this cluster: 1
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Reprocess disabled,
checking new event logs only
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - WARNING - Exception
opening file /home/toucan/pbslog00021123.gz
java.io.FileNotFoundException: /home/toucan/pbslog00021123.gz (No such
file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at
org.glite.apel.pbs.EventLogParser.<init>(EventLogParser.java:88)
at
org.glite.apel.pbs.EventLogProcessor.setupLogFiles(EventLogProcessor.jav
a:78)
at
org.glite.apel.pbs.EventLogProcessor.<init>(EventLogProcessor.java:57)
at
org.glite.apel.pbs.ApelLogParser.runEventProcessor(ApelLogParser.java:12
5)
at org.glite.apel.pbs.ApelLogParser.run(ApelLogParser.java:75)
at org.glite.apel.pbs.ApelLogParser.main(ApelLogParser.java:274)
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - WARNING - Exception
opening file /home/toucan/pbslog00021100.log
java.io.FileNotFoundException: /home/toucan/pbslog00021100.log (No such
file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at
org.glite.apel.pbs.EventLogParser.<init>(EventLogParser.java:88)
at
org.glite.apel.pbs.EventLogProcessor.setupLogFiles(EventLogProcessor.jav
a:78)
at
org.glite.apel.pbs.EventLogProcessor.<init>(EventLogProcessor.java:57)
at
org.glite.apel.pbs.ApelLogParser.runEventProcessor(ApelLogParser.java:12
5)
at org.glite.apel.pbs.ApelLogParser.run(ApelLogParser.java:75)
at org.glite.apel.pbs.ApelLogParser.main(ApelLogParser.java:274)
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - **** Updating PBS
end event table (EventRecords) ****
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050222
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050223
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050224
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050225
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050226
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050227
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050228
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050301
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050302
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050303
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050304
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050305
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050306
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050307
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050308
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050309
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050310
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050311
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050312
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050313
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050314
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050315
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050316
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050317
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050318
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050319
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050320
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050321
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050322
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050323
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050324
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050325
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050326
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050327
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050328
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050329
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050330
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050331
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050401
<SNIP>Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring
already processed event log file: 20050812
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050813
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050814
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050815
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050816
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050817
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050818
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050819
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050820
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050821
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050822
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050823
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050824
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050825
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050826
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050827
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050829
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: 20050830
Thu Sep 1 02:05:01 UTC 2005: apel-pbs-log-parser - Processing
/var/spool/pbs/server_priv/accounting/20050831
Thu Sep 1 02:05:04 UTC 2005: apel-pbs-log-parser - Processing
/var/spool/pbs/server_priv/accounting/20050901
Thu Sep 1 02:05:04 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: pbslog00021123.gz
Thu Sep 1 02:05:04 UTC 2005: apel-pbs-log-parser - Ignoring already
processed event log file: pbslog00021100.log
Thu Sep 1 02:05:04 UTC 2005: apel-pbs-log-parser - Event records
inserted: 1615
Thu Sep 1 02:05:04 UTC 2005: apel-pbs-log-parser - **** Updating Gate
Keeper log table (GkRecords) ****
Thu Sep 1 02:05:04 UTC 2005: apel-pbs-log-parser - Processing gate
keeper log file: /var/log/globus-gatekeeper.log
Thu Sep 1 02:05:09 UTC 2005: apel-pbs-log-parser - Already processed GK
log file: globus-gatekeeper.log.20050821040207.0
Thu Sep 1 02:05:09 UTC 2005: apel-pbs-log-parser - Already processed GK
log file: globus-gatekeeper.log.20050828040201.1
Thu Sep 1 02:05:09 UTC 2005: apel-pbs-log-parser - Gk records inserted:
1678
Thu Sep 1 02:05:09 UTC 2005: apel-pbs-log-parser - Msg records
inserted: 0
Thu Sep 1 02:05:09 UTC 2005: apel-pbs-log-parser - **** Updating Job ID
table (MessageRecords) from system message log ****
Thu Sep 1 02:05:09 UTC 2005: apel-pbs-log-parser - Processing message
log file: /var/log/messages
Thu Sep 1 02:05:09 UTC 2005: apel-pbs-log-parser - Processing message
log file: /var/log/messages.1
Thu Sep 1 02:05:09 UTC 2005: apel-pbs-log-parser - Processing message
log file: /var/log/messages.2
Thu Sep 1 02:05:12 UTC 2005: apel-pbs-log-parser - Processing message
log file: /var/log/messages.3
Thu Sep 1 02:05:16 UTC 2005: apel-pbs-log-parser - Processing message
log file: /var/log/messages.4
Thu Sep 1 02:05:19 UTC 2005: apel-pbs-log-parser - Msg records
inserted: 0
Thu Sep 1 02:05:19 UTC 2005: apel-pbs-log-parser - ------ Processing
finished ------
I have found a reference to setting APEL_HOME=/opt/glite on the mailing
list, this is set in /etc/cron.d/edg-apel-pbs-parser (entry below)
5 3 * * * root env RGMA_HOME=/opt/glite APEL_HOME=/opt/glite
/opt/glite/bin/apel-pbs-log-parser -f
/opt/glite/etc/glite-apel-pbs/parser-config-yaim.xml >>
/var/log/apel.log 2>&1
However, there is also a crontab entry in the root crontab (crontab -l)
PATH=/sbin:/bin:/usr/sbin:/usr/bin
29 1,7,13,19 * * * /opt/edg/sbin/edg-mkgridmap
- --output=/etc/grid-security/grid-mapfile --safe
0 5 * * * /opt/edg/sbin/lcg-expiregridmapdir.pl -v
1>>/var/log/lcg-expiregridmapdir.log 2>&1
5 3 * * * /opt/edg/sbin/edg-rgma-apel -f /opt/edg/etc/edg-rgma/apel.xml
>> /var/log/apel.log 2>&1
Has this been left over from the 2.4.0 install?
Any ideas
Mark.
- -------------------------------------------------------------
Mark Nelson - [log in to unmask]
IPPP, Department of Physics, University of Durham,
Science Laboratories, South Road, Durham, DH1 3LE
Office: +44 (0)191 334 3811, Direct Dial: +44 (0)191 334 3653
PGP Key: http://www.ippp.dur.ac.uk/~mn/pgp_key.txt
This mail is for the addressee only
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org
iD8DBQFDFtoIlzM++u0MgcERAiVnAJ9C/+4y5J2gYJSX8gBqM4Fvv4dVIgCfa6+K
2L/uqtkp20wnZ/ODK5i4nMQ=
=KLF2
-----END PGP SIGNATURE-----
|
