Hi Daniela
I'm checling /var/log/apel/log at ce's:
cat /var/log/apel.log
Sun Sep 22 00:35:07 UTC 2013: apel-pbs-log-parser - Blahd records inserted: 5106
Sun Sep 22 00:35:07 UTC 2013: apel-pbs-log-parser - ------ Processing finished ------
Mon Sep 23 00:35:01 UTC 2013: apel-pbs-log-parser - Read-in configuration: [quiet, p, inspectTables, c, e, recursiveEventFiles, b, recursiveBlahdFiles] [DBPassword=****, site=UKI-NORTHGRID-SHEF-HEP, pbslog=/var/torque/server_priv/accounting, GIIS=lcgce3.shef.ac.uk, CEhost=lcgce3.shef.ac.uk, blahdlogPrefix=blahp.log-, DBURL=jdbc:mysql://lcg.shef.ac.uk:3306/accounting, DBUsername=accounting, timeZone=UTC, blahdlog=/var/log/cream/accounting]
Mon Sep 23 00:35:01 UTC 2013: apel-pbs-log-parser - ------ Starting the apel application ------
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - **** APEL is examining the schema ****
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Checking the LcgRecords table
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The LcgRecords schema is up-to-date
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Checking the BlahdRecords table
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The BlahdRecords schema is up-to-date
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Checking the LcgProcessedFiles table
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The LcgProcessedFiles schema is up-to-date
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Checking the SpecRecords table for patch 28593
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Checking the SpecRecords table for patch 65723
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The SpecRecords schema is up-to-date
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Checking the GkRecords table
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The GkRecords schema is up-to-date
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Checking the MessageRecords table
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The MessageRecords schema is up-to-date
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - **** Schema checks complete ****
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Finding CPU spec values from GIIS server: lcgce3.shef.ac.uk
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - LDAP URL: ldap://lcgce3.shef.ac.uk:2170
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-ops:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-t2k:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-camont:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-alice:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-lhcb:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-minos:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-northgrid:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-dteam:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-snoplus:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-cms:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-geant4:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-pheno:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-ngs:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-sl6:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-gridpp:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-biomed:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required lcgce3.shef.ac.uk:8443/cream-pbs-mice:2995
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Reprocess disabled, checking new event logs only
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - **** Updating PBS end event table (EventRecords) ****
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Event records inserted: 0
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Checking the BlahdRecords table
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - The BlahdRecords schema is up-to-date
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Reprocess disabled, checking new event logs only
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Ignoring already processed accounting log file: lcgce3.shef.ac.uk /var/log/cream/accounting/blahp.log-20130123
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Ignoring already processed accounting log file: lcgce3.shef.ac.uk /var/log/cream/accounting/blahp.log-20130131
Mon Sep 23 00:35:02 UTC 2013: apel-pbs-log-parser - Ignoring already processed accounting log file: lcgce3.shef.ac.uk /var/log/cream/accounting/bl
Mon Sep 23 00:35:03 UTC 2013: apel-pbs-log-parser - Ignoring already processed accounting log file: lcgce3.shef.ac.uk /var/log/cream/accounting/blahp.log-20130818
Mon Sep 23 00:35:03 UTC 2013: apel-pbs-log-parser - Ignoring already processed accounting log file: lcgce3.shef.ac.uk /var/log/cream/accounting/blahp.log-20130304
Mon Sep 23 00:35:03 UTC 2013: apel-pbs-log-parser - Processing /var/log/cream/accounting/blahp.log-20130922
Mon Sep 23 00:35:07 UTC 2013: apel-pbs-log-parser - Ignoring already processed accounting log file: lcgce3.shef.ac.uk /var/log/cream/accounting/blahp.log-20130209
......
Mon Sep 23 00:35:07 UTC 2013: apel-pbs-log-parser - Ignoring already processed accounting log file: lcgce3.shef.ac.uk /var/log/cream/accounting/blahp.log-20130618
Mon Sep 23 00:35:07 UTC 2013: apel-pbs-log-parser - Blahd records inserted: 6374
Mon Sep 23 00:35:07 UTC 2013: apel-pbs-log-parser - ------ Processing finished ------
[root@lcgce3 ~]#
Thank you very much for your help
Elena
On 23 Sep 2013, at 14:45, Daniela Bauer wrote:
> Hi Elena,
>
> this looks like your Event Records stop on 27. August.
> The Event Records are generated on the CE by reading the batch system
> accounting files. Maybe the CE lost access to these files ? You need
> to check the APEL log on the CE(s) to see where it's going wrong
> (search for "Event records inserted" and "Blahd records inserted" -
> they should both be decidedly non-zero).
>
> Cheers,
> Daniela
>
> On 23 September 2013 14:39, Elena Korolkova <[log in to unmask]> wrote:
>> Hello
>>
>> I have a problem with publishing after certificate renewal (email address was removed from CN). Now this problem is fixed by changing certificate CN in goc. I managed to publish a week after but after this I have a problem:
>>
>>
>> [root@lcg glite-apel-publisher]# env APEL_HOME=/ JAVA_HOME=/etc/alternatives/java_sdk_1.6.0 /usr/bin/apel-publisher -f /etc/glite-apel-publisher/publisher-config-yaim.xml
>> =====Starting APEL Publisher=====
>> Mon Sep 23 12:30:15 BST 2013
>> Current versions of APEL RPMS:
>> glite-apel-core-2.0.14-5.sl5.noarch
>> glite-apel-yaim-1.0.2-4.sl5.noarch
>> emi-apel-1.0.0-0.sl5.noarch
>> glite-apel-publisher-2.0.16-1.sl5.noarch
>> Copying certificates from /etc/grid-security to the Java key store used by the APEL Publisher
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - Read-in configuration: [logenabled, p, inspectTables, j, publishGlobalDN] [DBUsername=accounting, DBURL=jdbc:mysql://localhost:3306/accounting, DBPassword=****, site=UKI-NORTHGRID-SHEF-HEP, Timeout for awaiting a reply from the consumer.=1800000, The max. num of accounting records in each JMS message=1000, Batch size=100000, republish=gap]
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - ------ Starting the APEL Publisher ------
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - **** APEL is examining the schema ****
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - Checking the LcgRecords table
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - The LcgRecords schema is up-to-date
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - Checking the BlahdRecords table
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - The BlahdRecords schema is up-to-date
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - Checking the LcgProcessedFiles table
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - The LcgProcessedFiles schema is up-to-date
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - Checking the SpecRecords table for patch 28593
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - Checking the SpecRecords table for patch 65723
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - The SpecRecords schema is up-to-date
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - Checking the GkRecords table
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - The GkRecords schema is up-to-date
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - Checking the MessageRecords table
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - The MessageRecords schema is up-to-date
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - **** Schema checks complete ****
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - **** Running the APEL publisher in Republish Gap Mode ****
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - Checking valid CPU spec data exists
>> Mon Sep 23 11:31:59 UTC 2013: apel-publisher - CPU spec values found
>> Mon Sep 23 11:33:12 UTC 2013: apel-publisher - Publishing 0 records...
>> Mon Sep 23 11:33:12 UTC 2013: apel-publisher - Publishing data into ActiveMQ Broker
>> Mon Sep 23 11:33:12 UTC 2013: apel-publisher - All UserDNs will be encrypted
>> Mon Sep 23 11:33:13 UTC 2013: apel-publisher - SecureRandom provider is SUN version 1.6
>> Mon Sep 23 11:33:13 UTC 2013: apel-publisher - Salting SecureRandom (SHA1PRNG) from /dev/urandom
>> KEY LOCATION =
>> Mon Sep 23 11:33:13 UTC 2013: apel-publisher - Updating MeasurementDate and MeasurementTime timestamp of Gap Data
>> Mon Sep 23 11:33:13 UTC 2013: apel-publisher - Total records published to Server with GAP Publisher:0
>> Mon Sep 23 11:33:13 UTC 2013: apel-publisher - ------ APEL Publisher finished ------
>>
>>
>> mysql> select Max(EventDate) from LcgRecords;
>> +----------------+
>> | Max(EventDate) |
>> +----------------+
>> | 2013-08-27 |
>> +----------------+
>> 1 row in set (11.19 sec)
>>
>> mysql> select Max(EventDate) from EventRecords;
>> +----------------+
>> | Max(EventDate) |
>> +----------------+
>> | 2013-08-27 |
>> +----------------+
>> 1 row in set (1 min 38.47 sec)
>>
>> mysql> select Max(ValidFrom) from BlahdRecords;
>> +----------------+
>> | Max(ValidFrom) |
>> +----------------+
>> | 2013-09-22 |
>> +----------------+
>> 1 row in set (1 min 43.20 sec)
>>
>> cat /etc/glite-apel-publisher/publisher-config-yaim.xml
>> <?xml version="1.0" encoding="UTF-8"?>
>>
>> <ApelConfiguration enableDebugLogging="yes">
>>
>> <SiteName>UKI-NORTHGRID-SHEF-HEP</SiteName>
>> <DBURL>jdbc:mysql://localhost:3306/accounting</DBURL>
>> <DBUsername>accounting</DBUsername>
>> <DBPassword>***</DBPassword>
>> <!-- Number of records selected each time
>> Modify this value if OutOfMemory error
>> appears in the Publisher
>> approx 150000 records per 512Mb memory -->
>> <Limit>100000</Limit>
>> <MaxInsertBatch>1000</MaxInsertBatch>
>> <!-- Timeout for waiting for consumer in milliseconds -->
>> <ConsumerTimeout>1800000</ConsumerTimeout>
>> <!-- Examine APEL Schema //-->
>> <DBProcessor inspectTables="yes"/>
>>
>> <!--
>> Publisher Options
>> =================
>> publishGlobalUserName="yes" : encrypt UserDNs with a 1024-bit RSA key
>> : If "no", UserDNs are not published (default)
>> publicKeyLocation="/home/key/public.der" : if set, this external public key
>> will be used for encryption. Otherwise APEL
>> key will be used
>> <Republish>missing</Republish> : publish missing data only (default)
>> <Republish>missing</Republish> : publish ALL accounting data in the database
>> Gap Publisher : publish data in a specified interval
>> <Republish recordStart="2006-02-01" recordEnd="2006-04-25">gap</Republish>
>> -->
>> <JoinProcessor publishGlobalUserName="yes">
>> <Republish recordStart="2013-08-27" recordEnd="2013-09-07">gap</Republish>
>> <!--
>> VO Filtering:
>> include="include" : Only the VOs specified in VOList will be published
>> include="exclude" : All VOs except the ones specified in VOList will be published
>> include="all" : All VOs will be published. VOList will be ignored.
>> Enter one VO per <VO> tag
>>
>> VO Mapping:
>> Map local unix groups to VOs if not using Blah Accounting Log files
>> <VOGroup>localgroup:VOName</VOGroup>
>> -->
>> <VOInclude include="all">
>> <!--VOList>
>> <VO>atlas</VO>
>> <VO>cms</VO>
>> </VOList-->
>> <!--VOMapping>
>> <VOGroup>atlprd:atlas</VOGroup>
>> <VOGroup>cmsprod:cms</VOGroup>
>> </VOMapping-->
>> </VOInclude>
>>
>>
>> </JoinProcessor>
>>
>> </ApelConfiguration>
>>
>>
>>
>> Your help is much appreciated.
>>
>> Elena
>> __________________________________________________
>> Dr Elena Korolkova
>> Email: [log in to unmask]
>> Tel.: +44 (0)114 2223553
>> Fax: +44 (0)114 2223555
>> Department of Physics and Astronomy
>> University of Sheffield
>> Sheffield, S3 7RH, United Kingdom
>
>
>
> --
> Sent from the pit of despair
>
> -----------------------------------------------------------
> [log in to unmask]
> HEP Group/Physics Dep
> Imperial College
> London, SW7 2BW
> Tel: +44-(0)20-75947810
> http://www.hep.ph.ic.ac.uk/~dbauer/
__________________________________________________
Dr Elena Korolkova
Email: [log in to unmask]
Tel.: +44 (0)114 2223553
Fax: +44 (0)114 2223555
Department of Physics and Astronomy
University of Sheffield
Sheffield, S3 7RH, United Kingdom
|
