Hello,
We recently added to our site (HG-01-GRNET) support for the BIOMED VO.
This was done through the YAIM installation of LCG-2.3.0 (by adding to
site-info.def the appropriate entries for this VO).
Unfortunately, we realized today that jobs submitted from users
belonging to this VO fail to run. Going through system logs, we found that
the Biomed users are properly mapped to the pool accounts biomedNNN, and
their jobs are submitted to Torque. However, the jobs fail to run. For
example, this is from /var/spool/pbs/server_priv/accounting/20050118 from
our Computing element:
01/18/2005 14:51:29;Q;4126.ce01.isabella.grnet.gr;queue=biomed
01/18/2005 14:52:19;D;4126.ce01.isabella.grnet.gr;[log in to unmask]
Searching the logs a little further, we found that whenever a biomed job
is submitted, entries like the following appear in
/var/spool/pbs/mom_logs/20050118 on the WN that is trying to run the
job:
01/18/2005 14:51:31;0080; pbs_mom;Fil;sys_copy;command: /bin/cp -r /gpfs1/biom
ed/biomed001/.lcgjm/globus-cache-export.4q706h/globus-cache-export.4q706h.gpg gl
obus-cache-export.4q706h.gpg status=1 (copy request failed), try=1
01/18/2005 14:51:31;0080; pbs_mom;Fil;sys_copy;command: /bin/cp -r /gpfs1/biom
ed/biomed001/.lcgjm/globus-cache-export.4q706h/globus-cache-export.4q706h.gpg gl
obus-cache-export.4q706h.gpg status=1 (copy request failed), try=2
01/18/2005 14:51:31;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=
0, type=11, from [log in to unmask]
01/18/2005 14:51:35;0080; pbs_mom;Fil;sys_copy;command: /bin/cp -r /gpfs1/biom
ed/biomed001/.lcgjm/globus-cache-export.4q706h/globus-cache-export.4q706h.gpg gl
obus-cache-export.4q706h.gpg status=1 (copy request failed), try=3
01/18/2005 14:51:35;0080; pbs_mom;Fil;sys_copy;command: /bin/cp -r /gpfs1/biom
ed/biomed001/.lcgjm/globus-cache-export.4q706h/globus-cache-export.4q706h.gpg gl
obus-cache-export.4q706h.gpg status=1 (copy request failed), try=4
01/18/2005 14:51:42;0004; pbs_mom;Fil;globus-cache-export.4q706h.gpg;Unable to
copy file globus-cache-export.4q706h.gpg from ce01.isabella.grnet.gr
01/18/2005 14:51:42;0004; pbs_mom;Fil;globus-cache-export.4q706h.gpg;/bin/cp:
cannot read symbolic link `/gpfs1/biomed/biomed001/.lcgjm/globus-cache-export.4q
706h/globus-cache-export.4q706h.gpg': Numerical result out of range
01/18/2005 14:51:42;0008; pbs_mom;Req;del_files;cannot stat globus-cache-expor
t.4q706h.gpg
After going into the directory in which the above errors were produced
and running 'ls -li' we came across the following errors:
[root@ce01 globus-cache-export.4q706h]# pwd
/gpfs1/biomed/biomed001/.lcgjm/globus-cache-export.4q706h
[root@ce01 globus-cache-export.4q706h]# ls -li
ls: cannot read symbolic link globus-cache-export.4q706h.gpg: Numerical result out of range
ls: cannot read symbolic link export.1: Numerical result out of range
ls: cannot read symbolic link export.2: Numerical result out of range
ls: cannot read symbolic link export.3: Numerical result out of range
ls: cannot read symbolic link export.4: Numerical result out of range
ls: cannot read symbolic link export.5: Numerical result out of range
total 88
917654 -rw-r--r-- 1 biomed001 biomed 20480 Jan 18 14:51 cache_export_dir.tar
942686 lrwxrwxrwx 1 biomed001 biomed 129 Jan 18 14:51 export.1
31583 lrwxrwxrwx 1 biomed001 biomed 129 Jan 18 14:51 export.2
591468 lrwxrwxrwx 1 biomed001 biomed 129 Jan 18 14:51 export.3
917658 lrwxrwxrwx 1 biomed001 biomed 129 Jan 18 14:51 export.4
612893 lrwxrwxrwx 1 biomed001 biomed 129 Jan 18 14:51 export.5
31550 -rw-r--r-- 1 biomed001 biomed 1303 Jan 18 14:51 export.txt
867334 -rw-r--r-- 1 biomed001 biomed 0 Jan 18 14:51 file_cleanup.txt
917584 lrwxrwxrwx 1 biomed001 biomed 129 Jan 18 14:51 globus-cache-export.4q706h.gpg
889349 -rw-r--r-- 1 biomed001 biomed 0 Jan 18 14:51 stage_in.txt
31482 -rw-r--r-- 1 biomed001 biomed 0 Jan 18 14:51 stage_out.txt
738450 -rw-r--r-- 1 biomed001 biomed 260 Jan 18 14:51 stdstreams.txt
Keep in mind that such errors appear only for biomed users. We have
tried almost every other VO supported by our site and there is no
problem...
Any ideas?
Thanks in advance...
--
Kyriakos Ginis
|