Hi all,

 

The PYTHONPATH at nodes is

 

/opt/glite/lib/python:/opt/lcg/lib64/python:/opt/lcg/lib64/python2.4/site-packages

 

Finally the srmv logs at se returns

 

GSS Minor Status Error Chain:

globus_gsi_gssapi: Error during delegation: Delegation protocol violation

 

06/19 19:23:19.403 30584,0 srmv1: SRM02 - soap_serve error : [::ffff:195.7.114.102] (fedex2.iscpif.fr) : Method 'ns1:srmPing' not implemented: method name or namespace not recognized

06/19 19:23:20.768 30584,1 srmv1: SRM02 - soap_serve error : [::ffff:195.7.114.102] (fedex2.iscpif.fr) : Method 'ns1:srmLs' not implemented: method name or namespace not recognized

06/19 19:23:22.621 30584,0 srmv1: SRM02 - soap_serve error : [::ffff:195.7.114.102] (fedex2.iscpif.fr) : Method 'ns1:srmPing' not implemented: method name or namespace not recognized

06/19 19:23:23.152 30584,0 srmv1: SRM02 - soap_serve error : [::ffff:195.7.114.102] (fedex2.iscpif.fr) : Method 'ns1:srmLs' not implemented: method name or namespace not recognized

 

Thanks

 

Konstantinos Koumoutsos

 

From: LHC Computer Grid - Rollout [mailto:[log in to unmask]] On Behalf Of Christos Triantafyllidis
Sent: Friday, June 17, 2011 11:49 AM
To: [log in to unmask]
Subject: Re: [LCG-ROLLOUT] OPS jobs delay to be executed

 

BTW the nagios alarm you are having can be found on the org.sam.WN-RepCr test and for this case it is the following. It appears that lfc python libraries have been changed or a PYTHONPATH is not set correctly at your nodes. Could you check if you can copy register (lcg-cr) a file as a pool account at your WNs?

 

Regards,

Christos

 

wn044.kallisto.hellasgrid.gr: CRITICAL: File was NOT copied to SE se01.kallisto.hellasgrid.gr and registered in LFC prod-lfc-shared-central.cern.ch. [ErrDB:[('lcg_util_wn', 'server', 'CRITICAL')]] CLI
CRITICAL: File was NOT copied to SE se01.kallisto.hellasgrid.gr and registered in LFC prod-lfc-shared-central.cern.ch. [ErrDB:[('lcg_util_wn', 'server', 'CRITICAL')]] CLI
Testing from: wn044.kallisto.hellasgrid.gr
DN: /C=GR/O=HellasGrid/OU=auth.gr/CN=Christos Triantafyllidis/CN=proxy/CN=proxy/CN=proxy/CN=proxy/CN=limited proxy
VOMS FQANs: /ops/NGI/Greece/Role=NULL/Capability=NULL, /ops/NGI/Role=NULL/Capability=NULL, /ops/Role=NULL/Capability=NULL, /ops/ROC/Role=NULL/Capability=NULL
Check if we can write to LFC prod-lfc-shared-central.cern.ch
lcg_util-1.7.6-2
GFAL-client-1.11.8-3
Copy file to SE and register in LFC.
2011-06-17T08:21:56Z
LFC: prod-lfc-shared-central.cern.ch
Using CLI:
lcg-cr -v --srm-timeout 180 --connect-timeout 10 --sendreceive-timeout 120 --bdii-timeout 20 --vo ops -d se01.kallisto.hellasgrid.gr -l lfn:/grid/ops/SAM/sam-lcg-rm-cr-wn044.kallisto.hellasgrid.gr.110617082156.9928965 /home/ops/ops124/gram_scratch_vq7aNGVS5S/https_3a_2f_2flb01.egee-see.org_3a9000_2fb7BWO_5fFu-8T_5f7h8XMgKOeQ/gridprobes/ops.NGI.Greece/org.sam/WN/localhost.localdomain/testFile.txt

Using grid catalog type: lfc 
Using grid catalog : (null) 
Checksum type: None 
SE type: SRMv2 
Destination SURL : srm://se01.kallisto.hellasgrid.gr/dpm/kallisto.hellasgrid.gr/home/ops/generated/2011-06-17/file41c89380-939c-4b35-86f6-3dc571ff4406 
Source SRM Request Token: a4b69f2b-a71a-49bd-9cdc-3999caddffae 
Source URL: file:/home/ops/ops124/gram_scratch_vq7aNGVS5S/https_3a_2f_2flb01.egee-see.org_3a9000_2fb7BWO_5fFu-8T_5f7h8XMgKOeQ/gridprobes/ops.NGI.Greece/org.sam/WN/localhost.localdomain/testFile.txt 
File size: 240 
VO name: ops 
Destination specified: se01.kallisto.hellasgrid.gr 
Destination URL for copy: gsiftp://se01.kallisto.hellasgrid.gr/se01.kallisto.hellasgrid.gr:/data02/ops/2011-06-17/file41c89380-939c-4b35-86f6-3dc571ff4406.2076330.0 
# streams: 1 
0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 240 bytes 1.12 KB/sec avg 1.12 KB/sec inst 
Transfer took 1020 ms 
[GFAL][lfc_init][] liblfc.so: liblfc.so: cannot open shared object file: No such file or directory 
srm://se01.kallisto.hellasgrid.gr/dpm/kallisto.hellasgrid.gr/home/ops/generated/2011-06-17/file41c89380-939c-4b35-86f6-3dc571ff4406: Registration failed, please register it by hand, when the problem will be solved 
guid:fc858ffb-7540-487a-aead-dcbaa88493f1 
lcg_cr: Communication error on send 

2011-06-17T08:22:04Z

On Jun 17, 2011, at 11:21 AM, Gkamas Vasilis wrote:

Dear all,

At our site, we face a problem with ops jobs which started when we updated to glite-WN 3.2.11-0. The site has 100% utilization all that days but while ops jobs are waiting to be executed, jobs from other VOs are firstly executed despite the fact that these jobs arrive to site afterward that ops jobs, which results in low site availability. I attach the maui.cfg file.

 

I send you also the description of a nagios alarm but I am not sure if this error is related with the above problem.

 

CRITICAL: METRIC FAILED [org.sam.WN-RepCr]: CRITICAL: File was NOT copied to SE se01.kallisto.hellasgrid.gr and registered in LFC prod-lfc-shared-central.cern.ch. [ErrDB:[('lcg_util_wn', 'server', 'CRITICAL')]] CLI

Testing from: wn011.kallisto.hellasgrid.gr

DN: /C=GR/O=HellasGrid/OU=auth.gr/CN=Christos Triantafyllidis/CN=proxy/CN=proxy/CN=proxy/CN=proxy/CN=limited proxy

VOMS FQANs: /ops/NGI/Greece/Role=NULL/Capability=NULL, /ops/NGI/Role=NULL/Capability=NULL, /ops/Role=NULL/Capability=NULL, /ops/ROC/Role=NULL/Capability=NULL

Invoking metric: [2011-06-16T20:41:36Z] org.sam.WN-RepISenv

Invoking metric: [2011-06-16T20:41:36Z] org.sam.WN-RepFree

Invoking metric: [2011-06-16T20:41:36Z] org.sam.WN-RepCr

METRIC FAILED [org.sam.WN-RepCr]: CRITICAL: File was NOT copied to SE se01.kallisto.hellasgrid.gr and registered in LFC prod-lfc-shared-central.cern.ch. [ErrDB:[('lcg_util_wn', 'server', 'CRITICAL')]] CLI

 

Has anyone faced a problem like the above?

 

Thank you,

 

Gkamas Vasileios, HG-04-CTI-CEID admin team

 

---------------------------------------------------------

Gkamas Vasileios, MSc

Computer Engineer and Informatics

Networking Technologies Sector

Research Academic Computer Technology Institute

N. Kazantzaki, Univeristy of Patras, 26500, Rion, Greece

Tel. +30 2610 960 408, Fax. +30 2610 960 350, Mob. +30 6977 51 52 22

email: [log in to unmask]

---------------------------------------------------------

 

<maui.cfg>