Hi,
I 've just tried to submit a simple job but it finished aborted :-(
From the logging-info:
---
Event: Done
- exit_code = 1
- host = egee-rb-01.cnaf.infn.it
- level = SYSTEM
- priority = asynchronous
- reason = Got a job held event, reason: Unspecified
gridmanager error
- seqcode =
UI=000003:NS=0000000003:WM=000016:BH=0000000000:JSS=000012:LM=000031:LRMS=000000:APP=000000
- source = LogMonitor
- src_instance = unique
- status_code = FAILED
- timestamp = Thu Nov 24 09:02:21 2005
- user = /C=IT/O=INFN/OU=Personal
Certificate/L=CNAF/CN=alessandro
[log in to unmask]
---
Event: Done
- exit_code = 1
- host = egee-rb-01.cnaf.infn.it
- level = SYSTEM
- priority = asynchronous
- reason = Job got an error while in the CondorG queue.
- seqcode =
UI=000003:NS=0000000003:WM=000016:BH=0000000000:JSS=000012:LM=000033:LRMS=000000:APP=000000
- source = LogMonitor
- src_instance = unique
- status_code = FAILED
- timestamp = Thu Nov 24 09:02:32 2005
- user = /C=IT/O=INFN/OU=Personal
Certificate/L=CNAF/CN=alessandro
[log in to unmask]
---
So it seems to be ssh problems between CE and WNs, but also it could be
some CRLs out of date.
When i launched theese commands:
[paolini@lcg-ui paolini]$ globus-job-run
CE.pakgrid.org.pk/jobmanager-lcgpbs -queue dteam /bin/hostname
[paolini@lcg-ui paolini]$ globus-job-run
CE.pakgrid.org.pk/jobmanager-lcgpbs /bin/hostname
no answer returned.
CE identifies me as dteam006.
Take a look to theese faqs:
http://goc.grid.sinica.edu.tw/gocwiki/submit-helper_script_%2e%2e%2e_gave_error%3a_cache_export_dir_%2e%2e%2e
http://goc.grid.sinica.edu.tw/gocwiki/ssh_problem_from_WN_to_CE
Cheers,
Alessandro
Asif Osman ha scritto:
>Hi,
>
>We are getting JS error in goc database.
>Reason seems to be authentication failure between WN to CE communication.
>Sequence of events occuring are as follows:
>1) jobs submitted from UI
>2) It is landed on the CE.pakgrid.org.pk
>3) CE submits this job to WN, where it is executed without any problem
>4) When job finishes, WN tries to copy stderr and stdout files to CE
>
>At this stage, the authentication fails and job is aborted.
>
>Following is the error messages logged in file:
>/var/spool/pbs/mom_logs/20051124
>
>============================================================================================================
>
>11/24/2005 10:17:04;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/dteam001/.lcgjm/globus-cache-export.k22183/globus-cache-export.k22183.gpg globus-cache-export.k22183.gpg status=1 (copy request failed), try=311/24/2005 10:17:35;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/dteam001/.lcgjm/globus-cache-export.k22183/globus-cache-export.k22183.gpg globus-cache-export.k22183.gpg status=1 (copy request failed), try=4
>11/24/2005 10:17:42;0004; pbs_mom;Fil;globus-cache-export.k22183.gpg;Unable to copy file globus-cache-export.k22183.gpg from CE.pakgrid.org.pk:/home/dteam001/.lcgjm/globus-cache-export.k22183/globus-cache-export.k22183.gpg
>11/24/2005 10:17:42;0004; pbs_mom;Fil;globus-cache-export.k22183.gpg;CE.pakgrid.org.pk: Connection refused
>11/24/2005 10:17:42;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.k22183.gpg
>11/24/2005 10:17:46;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 10:17:46;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.U22611/globus-cache-export.U22611.gpg globus-cache-export.U22611.gpg status=1 (copy request failed), try=1
>11/24/2005 10:18:17;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.U22611/globus-cache-export.U22611.gpg globus-cache-export.U22611.gpg status=1 (copy request failed), try=2
>11/24/2005 10:18:21;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.U22611/globus-cache-export.U22611.gpg globus-cache-export.U22611.gpg status=1 (copy request failed), try=3
>11/24/2005 10:18:52;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.U22611/globus-cache-export.U22611.gpg globus-cache-export.U22611.gpg status=1 (copy request failed), try=4
>11/24/2005 10:18:59;0004; pbs_mom;Fil;globus-cache-export.U22611.gpg;Unable to copy file globus-cache-export.U22611.gpg from CE.pakgrid.org.pk:/home/lhcb001/.lcgjm/globus-cache-export.U22611/globus-cache-export.U22611.gpg
>11/24/2005 10:18:59;0004; pbs_mom;Fil;globus-cache-export.U22611.gpg;CE.pakgrid.org.pk: Connection refused
>11/24/2005 10:18:59;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.U22611.gpg
>11/24/2005 10:22:48;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 10:22:48;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.Q24347/globus-cache-export.Q24347.gpg globus-cache-export.Q24347.gpg status=1 (copy request failed), try=1
>11/24/2005 10:23:19;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.Q24347/globus-cache-export.Q24347.gpg globus-cache-export.Q24347.gpg status=1 (copy request failed), try=2
>11/24/2005 10:23:23;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.Q24347/globus-cache-export.Q24347.gpg globus-cache-export.Q24347.gpg status=1 (copy request failed), try=3
>11/24/2005 10:23:54;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.Q24347/globus-cache-export.Q24347.gpg globus-cache-export.Q24347.gpg status=1 (copy request failed), try=4
>11/24/2005 10:24:01;0004; pbs_mom;Fil;globus-cache-export.Q24347.gpg;Unable to copy file globus-cache-export.Q24347.gpg from CE.pakgrid.org.pk:/home/lhcb001/.lcgjm/globus-cache-export.Q24347/globus-cache-export.Q24347.gpg
>11/24/2005 10:24:01;0004; pbs_mom;Fil;globus-cache-export.Q24347.gpg;CE.pakgrid.org.pk: Connection refused
>11/24/2005 10:24:01;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.Q24347.gpg
>11/24/2005 11:04:18;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 11:04:18;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.gi1902/globus-cache-export.gi1902.gpg globus-cache-export.gi1902.gpg status=1 (copy request failed), try=1
>11/24/2005 11:04:49;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.gi1902/globus-cache-export.gi1902.gpg globus-cache-export.gi1902.gpg status=1 (copy request failed), try=211/24/2005 11:04:53;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.gi1902/globus-cache-export.gi1902.gpg globus-cache-export.gi1902.gpg status=1 (copy request failed), try=3
>11/24/2005 11:05:24;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.gi1902/globus-cache-export.gi1902.gpg globus-cache-export.gi1902.gpg status=1 (copy request failed), try=411/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;Unable to copy file globus-cache-export.gi1902.gpg from CE.pakgrid.org.pk:/home/cms001/.lcgjm/globus-cache-export.gi1902/globus-cache-export.gi1902.gpg
>11/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;CE.pakgrid.org.pk: Connection refused
>11/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;lure
>11/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;No such file or directory
>11/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;
>11/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;Permission denied (external-keyx,gssapi,publickey,password,keyboard-interactive).
>11/24/2005 11:05:31;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.gi1902.gpg
>11/24/2005 11:09:22;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 11:09:22;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.xG3306/globus-cache-export.xG3306.gpg globus-cache-export.xG3306.gpg status=1 (copy request failed), try=1
>11/24/2005 11:09:53;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.xG3306/globus-cache-export.xG3306.gpg globus-cache-export.xG3306.gpg status=1 (copy request failed), try=211/24/2005 11:09:57;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.xG3306/globus-cache-export.xG3306.gpg globus-cache-export.xG3306.gpg status=1 (copy request failed), try=3
>11/24/2005 11:10:29;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.xG3306/globus-cache-export.xG3306.gpg globus-cache-export.xG3306.gpg status=1 (copy request failed), try=411/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;Unable to copy file globus-cache-export.xG3306.gpg from CE.pakgrid.org.pk:/home/cms001/.lcgjm/globus-cache-export.xG3306/globus-cache-export.xG3306.gpg
>11/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;CE.pakgrid.org.pk: Connection refused
>11/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;lure
>11/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;No such file or directory
>11/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;
>11/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;Permission denied (external-keyx,gssapi,publickey,password,keyboard-interactive).
>11/24/2005 11:10:36;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.xG3306.gpg
>11/24/2005 11:14:19;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 11:14:19;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.VZ4905/globus-cache-export.VZ4905.gpg globus-cache-export.VZ4905.gpg status=1 (copy request failed), try=1
>11/24/2005 11:14:50;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.VZ4905/globus-cache-export.VZ4905.gpg globus-cache-export.VZ4905.gpg status=1 (copy request failed), try=211/24/2005 11:14:54;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.VZ4905/globus-cache-export.VZ4905.gpg globus-cache-export.VZ4905.gpg status=1 (copy request failed), try=3
>11/24/2005 11:15:26;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.VZ4905/globus-cache-export.VZ4905.gpg globus-cache-export.VZ4905.gpg status=1 (copy request failed), try=411/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;Unable to copy file globus-cache-export.VZ4905.gpg from CE.pakgrid.org.pk:/home/cms001/.lcgjm/globus-cache-export.VZ4905/globus-cache-export.VZ4905.gpg
>11/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;CE.pakgrid.org.pk: Connection refused
>11/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;lure
>11/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;No such file or directory
>11/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;
>11/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;Permission denied (external-keyx,gssapi,publickey,password,keyboard-interactive).
>11/24/2005 11:15:33;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.VZ4905.gpg
>11/24/2005 11:19:21;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 11:19:21;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.JC6269/globus-cache-export.JC6269.gpg globus-cache-export.JC6269.gpg status=1 (copy request failed), try=1
>11/24/2005 11:19:52;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.JC6269/globus-cache-export.JC6269.gpg globus-cache-export.JC6269.gpg status=1 (copy request failed), try=211/24/2005 11:19:56;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.JC6269/globus-cache-export.JC6269.gpg globus-cache-export.JC6269.gpg status=1 (copy request failed), try=3
>11/24/2005 11:20:27;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.JC6269/globus-cache-export.JC6269.gpg globus-cache-export.JC6269.gpg status=1 (copy request failed), try=411/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;Unable to copy file globus-cache-export.JC6269.gpg from CE.pakgrid.org.pk:/home/cms001/.lcgjm/globus-cache-export.JC6269/globus-cache-export.JC6269.gpg
>11/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;CE.pakgrid.org.pk: Connection refused
>11/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;lure
>11/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;No such file or directory
>11/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;
>11/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;Permission denied (external-keyx,gssapi,publickey,password,keyboard-interactive).
>11/24/2005 11:20:34;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.JC6269.gpg
>
>Any solution?
>Cheers,
>Asif Osman
>
>
--
Dr. Alessandro Paolini
INFN - CNAF
Viale Berti Pichat 6/2
40127 Bologna
Italy
tel: +39 051 6092723
fax: +39 051 6092746
ICQ: 192172027
|