Hi,
We are getting JS error in goc database.
Reason seems to be authentication failure between WN to CE communication.
Sequence of events occuring are as follows:
1) jobs submitted from UI
2) It is landed on the CE.pakgrid.org.pk
3) CE submits this job to WN, where it is executed without any problem
4) When job finishes, WN tries to copy stderr and stdout files to CE
At this stage, the authentication fails and job is aborted.
Following is the error messages logged in file:
/var/spool/pbs/mom_logs/20051124
============================================================================================================
11/24/2005 10:17:04;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/dteam001/.lcgjm/globus-cache-export.k22183/globus-cache-export.k22183.gpg globus-cache-export.k22183.gpg status=1 (copy request failed), try=311/24/2005 10:17:35;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/dteam001/.lcgjm/globus-cache-export.k22183/globus-cache-export.k22183.gpg globus-cache-export.k22183.gpg status=1 (copy request failed), try=4
11/24/2005 10:17:42;0004; pbs_mom;Fil;globus-cache-export.k22183.gpg;Unable to copy file globus-cache-export.k22183.gpg from CE.pakgrid.org.pk:/home/dteam001/.lcgjm/globus-cache-export.k22183/globus-cache-export.k22183.gpg
11/24/2005 10:17:42;0004; pbs_mom;Fil;globus-cache-export.k22183.gpg;CE.pakgrid.org.pk: Connection refused
11/24/2005 10:17:42;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.k22183.gpg
11/24/2005 10:17:46;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 10:17:46;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.U22611/globus-cache-export.U22611.gpg globus-cache-export.U22611.gpg status=1 (copy request failed), try=1
11/24/2005 10:18:17;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.U22611/globus-cache-export.U22611.gpg globus-cache-export.U22611.gpg status=1 (copy request failed), try=2
11/24/2005 10:18:21;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.U22611/globus-cache-export.U22611.gpg globus-cache-export.U22611.gpg status=1 (copy request failed), try=3
11/24/2005 10:18:52;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.U22611/globus-cache-export.U22611.gpg globus-cache-export.U22611.gpg status=1 (copy request failed), try=4
11/24/2005 10:18:59;0004; pbs_mom;Fil;globus-cache-export.U22611.gpg;Unable to copy file globus-cache-export.U22611.gpg from CE.pakgrid.org.pk:/home/lhcb001/.lcgjm/globus-cache-export.U22611/globus-cache-export.U22611.gpg
11/24/2005 10:18:59;0004; pbs_mom;Fil;globus-cache-export.U22611.gpg;CE.pakgrid.org.pk: Connection refused
11/24/2005 10:18:59;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.U22611.gpg
11/24/2005 10:22:48;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 10:22:48;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.Q24347/globus-cache-export.Q24347.gpg globus-cache-export.Q24347.gpg status=1 (copy request failed), try=1
11/24/2005 10:23:19;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.Q24347/globus-cache-export.Q24347.gpg globus-cache-export.Q24347.gpg status=1 (copy request failed), try=2
11/24/2005 10:23:23;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.Q24347/globus-cache-export.Q24347.gpg globus-cache-export.Q24347.gpg status=1 (copy request failed), try=3
11/24/2005 10:23:54;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/lhcb001/.lcgjm/globus-cache-export.Q24347/globus-cache-export.Q24347.gpg globus-cache-export.Q24347.gpg status=1 (copy request failed), try=4
11/24/2005 10:24:01;0004; pbs_mom;Fil;globus-cache-export.Q24347.gpg;Unable to copy file globus-cache-export.Q24347.gpg from CE.pakgrid.org.pk:/home/lhcb001/.lcgjm/globus-cache-export.Q24347/globus-cache-export.Q24347.gpg
11/24/2005 10:24:01;0004; pbs_mom;Fil;globus-cache-export.Q24347.gpg;CE.pakgrid.org.pk: Connection refused
11/24/2005 10:24:01;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.Q24347.gpg
11/24/2005 11:04:18;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 11:04:18;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.gi1902/globus-cache-export.gi1902.gpg globus-cache-export.gi1902.gpg status=1 (copy request failed), try=1
11/24/2005 11:04:49;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.gi1902/globus-cache-export.gi1902.gpg globus-cache-export.gi1902.gpg status=1 (copy request failed), try=211/24/2005 11:04:53;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.gi1902/globus-cache-export.gi1902.gpg globus-cache-export.gi1902.gpg status=1 (copy request failed), try=3
11/24/2005 11:05:24;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.gi1902/globus-cache-export.gi1902.gpg globus-cache-export.gi1902.gpg status=1 (copy request failed), try=411/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;Unable to copy file globus-cache-export.gi1902.gpg from CE.pakgrid.org.pk:/home/cms001/.lcgjm/globus-cache-export.gi1902/globus-cache-export.gi1902.gpg
11/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;CE.pakgrid.org.pk: Connection refused
11/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;lure
11/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;No such file or directory
11/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;
11/24/2005 11:05:31;0004; pbs_mom;Fil;globus-cache-export.gi1902.gpg;Permission denied (external-keyx,gssapi,publickey,password,keyboard-interactive).
11/24/2005 11:05:31;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.gi1902.gpg
11/24/2005 11:09:22;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 11:09:22;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.xG3306/globus-cache-export.xG3306.gpg globus-cache-export.xG3306.gpg status=1 (copy request failed), try=1
11/24/2005 11:09:53;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.xG3306/globus-cache-export.xG3306.gpg globus-cache-export.xG3306.gpg status=1 (copy request failed), try=211/24/2005 11:09:57;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.xG3306/globus-cache-export.xG3306.gpg globus-cache-export.xG3306.gpg status=1 (copy request failed), try=3
11/24/2005 11:10:29;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.xG3306/globus-cache-export.xG3306.gpg globus-cache-export.xG3306.gpg status=1 (copy request failed), try=411/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;Unable to copy file globus-cache-export.xG3306.gpg from CE.pakgrid.org.pk:/home/cms001/.lcgjm/globus-cache-export.xG3306/globus-cache-export.xG3306.gpg
11/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;CE.pakgrid.org.pk: Connection refused
11/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;lure
11/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;No such file or directory
11/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;
11/24/2005 11:10:36;0004; pbs_mom;Fil;globus-cache-export.xG3306.gpg;Permission denied (external-keyx,gssapi,publickey,password,keyboard-interactive).
11/24/2005 11:10:36;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.xG3306.gpg
11/24/2005 11:14:19;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 11:14:19;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.VZ4905/globus-cache-export.VZ4905.gpg globus-cache-export.VZ4905.gpg status=1 (copy request failed), try=1
11/24/2005 11:14:50;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.VZ4905/globus-cache-export.VZ4905.gpg globus-cache-export.VZ4905.gpg status=1 (copy request failed), try=211/24/2005 11:14:54;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.VZ4905/globus-cache-export.VZ4905.gpg globus-cache-export.VZ4905.gpg status=1 (copy request failed), try=3
11/24/2005 11:15:26;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.VZ4905/globus-cache-export.VZ4905.gpg globus-cache-export.VZ4905.gpg status=1 (copy request failed), try=411/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;Unable to copy file globus-cache-export.VZ4905.gpg from CE.pakgrid.org.pk:/home/cms001/.lcgjm/globus-cache-export.VZ4905/globus-cache-export.VZ4905.gpg
11/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;CE.pakgrid.org.pk: Connection refused
11/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;lure
11/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;No such file or directory
11/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;
11/24/2005 11:15:33;0004; pbs_mom;Fil;globus-cache-export.VZ4905.gpg;Permission denied (external-keyx,gssapi,publickey,password,keyboard-interactive).
11/24/2005 11:15:33;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.VZ4905.gpg
11/24/2005 11:19:21;0080; pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=11, from [log in to unmask] 11:19:21;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.JC6269/globus-cache-export.JC6269.gpg globus-cache-export.JC6269.gpg status=1 (copy request failed), try=1
11/24/2005 11:19:52;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.JC6269/globus-cache-export.JC6269.gpg globus-cache-export.JC6269.gpg status=1 (copy request failed), try=211/24/2005 11:19:56;0080; pbs_mom;Fil;sys_copy;command: /usr/bin/scp -Br [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.JC6269/globus-cache-export.JC6269.gpg globus-cache-export.JC6269.gpg status=1 (copy request failed), try=3
11/24/2005 11:20:27;0080; pbs_mom;Fil;sys_copy;command: /usr/sbin/pbs_rcp -r [log in to unmask]:/home/cms001/.lcgjm/globus-cache-export.JC6269/globus-cache-export.JC6269.gpg globus-cache-export.JC6269.gpg status=1 (copy request failed), try=411/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;Unable to copy file globus-cache-export.JC6269.gpg from CE.pakgrid.org.pk:/home/cms001/.lcgjm/globus-cache-export.JC6269/globus-cache-export.JC6269.gpg
11/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;CE.pakgrid.org.pk: Connection refused
11/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;lure
11/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;No such file or directory
11/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;
11/24/2005 11:20:34;0004; pbs_mom;Fil;globus-cache-export.JC6269.gpg;Permission denied (external-keyx,gssapi,publickey,password,keyboard-interactive).
11/24/2005 11:20:34;0008; pbs_mom;Req;del_files;cannot stat globus-cache-export.JC6269.gpg
Any solution?
Cheers,
Asif Osman
|