Heya
>
> t2k004 is my t2k ID at Lancaster.
>
> Could you have words with your sudoers file ?
Gaahhh! I did this for ngs.ac.uk yesterday, I should have figured out
I'd need to do it for t2k too (as the two VOs had the same problem in
their yaim configuration).
I think I checked out for Christmas sometime early yesterday afternoon.
I'm just going to back away slowly from anything I could break horribly.
Although it should be fixed now. Or my CE has melted.
Thanks,
Matt
>
> Cheers,
>
>
>
>
> On 20 December 2012 11:32, Matt Doidge <[log in to unmask]
> <mailto:[log in to unmask]>> wrote:
>
> Hello again,
>
>
> I'll get back to Jon P to see if he sees any changes, along with
> Ewan's
> suggestion of trying direct job submission.
>
>
> Jon did a lot of investigating last night into the t2k problems at
> Lancaster, and found that direct job submission to our CE worked, as
> well as WMS submission from some WMSes. Other failed with the same
> failure mode (https://ggus.eu/ws/ticket___info.php?ticket=88628
> <https://ggus.eu/ws/ticket_info.php?ticket=88628>).
>
> The successful WMSi were:
> lcgwms02 & lcgwms03.gridpp.ac.uk <http://lcgwms03.gridpp.ac.uk> at RAL
>
> The unsuccessful WMS jobs went through:
> wms01.grid.hep.ph.ic.ac.uk <http://wms01.grid.hep.ph.ic.ac.uk> &
> wms02.grid.hep.ph.ic.ac.uk <http://wms02.grid.hep.ph.ic.ac.uk> (and
> lcgwms04 at RAL).
>
> The failed jobs all seemed to abort with the same long,
> authenticationy looking error message detailed previously: "error:
> globus_ftp_client: the server responded with an error500 500 ....
> Unable to open file ...Cannot move ISB".
>
> I don't know if there's anything significant that these WMS have in
> common? It could be that the gubbins that control the interaction
> between abaddon.hec.lancs.ac.uk <http://abaddon.hec.lancs.ac.uk> and
> these WMS has got into a bad state.
>
> Any ideas appreciated!
> Thanks,
> Matt
>
>
>
> Cheers,
> Matt
> P.S. Chris will be pleased that I had a bash at fixing ngs.ac.uk
> <http://ngs.ac.uk> access
> on our CE :-)
>
> On 12/19/2012 11:07 AM, Christopher J. Walker wrote:
>
> On 19/12/12 10:56, Daniela Bauer wrote:
>
> Hi Matt,
>
> check your CE:
>
> lx05:~ :~] voms-proxy-init -valid 24:00 --voms t2k.org
> <http://t2k.org> <http://t2k.org>
> Enter GRID pass phrase:
> Your identity:
> /C=UK/O=eScience/OU=Imperial/__L=Physics/CN=daniela bauer
> Creating temporary proxy
> ..............................__..............................__......
> Done
> Contacting voms.gridpp.ac.uk:15003
> <http://voms.gridpp.ac.uk:15003>
> <http://voms.gridpp.ac.uk:__15003
> <http://voms.gridpp.ac.uk:15003>>
> [/C=UK/O=eScience/OU=__Manchester/L=HEP/CN=voms.__gridpp.ac.uk
> <http://voms.gridpp.ac.uk>
> <http://voms.gridpp.ac.uk>] "t2k.org <http://t2k.org>
> <http://t2k.org>" Done
> Creating proxy
> ..............................__..............................__..............................__..............................
>
>
> Done
> Your proxy is valid until Thu Dec 20 10:51:18 2012
>
> lx05:~ :~] uberftp abaddon.hec.lancs.ac.uk
> <http://abaddon.hec.lancs.ac.uk>
> <http://abaddon.hec.lancs.ac.__uk
> <http://abaddon.hec.lancs.ac.uk>>
> 220 abaddon.hec.lancs.ac.uk
> <http://abaddon.hec.lancs.ac.uk>
> <http://abaddon.hec.lancs.ac.__uk
> <http://abaddon.hec.lancs.ac.uk>> GridFTP
> Server 6.10 (gcc64, 1334324800-83) [Globus Toolkit
> 5.2.0] ready.
> 530-Login incorrect. : globus_gss_assist: Error invoking
> callout
> 530-globus_callout_module: The callout returned an error
> 530-an unknown error occurred
> 530 End.
>
> But:
> lx05:~ :~] voms-proxy-init -valid 24:00 --voms dteam
> Enter GRID pass phrase:
> Your identity:
> /C=UK/O=eScience/OU=Imperial/__L=Physics/CN=daniela bauer
> Creating temporary proxy
> ..............................__......... Done
> Contacting voms2.hellasgrid.gr:15004
> <http://voms2.hellasgrid.gr:15004>
> <http://voms2.hellasgrid.gr:__15004
> <http://voms2.hellasgrid.gr:15004>>
> [/C=GR/O=HellasGrid/OU=hellasg__rid.gr/CN=voms2.hellasgrid.gr
> <http://hellasgrid.gr/CN=voms2.hellasgrid.gr>
> <http://hellasgrid.gr/CN=__voms2.hellasgrid.gr
> <http://hellasgrid.gr/CN=voms2.hellasgrid.gr>>] "dteam"
> Failed
>
> Error: Error during SSL handshake:
>
> Trying next server for dteam.
> Creating temporary proxy ....................... Done
> Contacting voms.hellasgrid.gr:15004
> <http://voms.hellasgrid.gr:15004>
> <http://voms.hellasgrid.gr:__15004
> <http://voms.hellasgrid.gr:15004>>
> [/C=GR/O=HellasGrid/OU=hellasg__rid.gr/CN=voms.hellasgrid.gr
> <http://hellasgrid.gr/CN=voms.hellasgrid.gr>
> <http://hellasgrid.gr/CN=voms.__hellasgrid.gr
> <http://hellasgrid.gr/CN=voms.hellasgrid.gr>>] "dteam" Done
> Creating proxy .......................... Done
> Your proxy is valid until Thu Dec 20 10:52:02 2012
>
>
> lx05:~ :~] uberftp abaddon.hec.lancs.ac.uk
> <http://abaddon.hec.lancs.ac.uk>
> <http://abaddon.hec.lancs.ac.__uk
> <http://abaddon.hec.lancs.ac.uk>>
> 220 abaddon.hec.lancs.ac.uk
> <http://abaddon.hec.lancs.ac.uk>
> <http://abaddon.hec.lancs.ac.__uk
> <http://abaddon.hec.lancs.ac.uk>> GridFTP
> Server 6.10 (gcc64, 1334324800-83) [Globus Toolkit
> 5.2.0] ready.
> 230 User dteam167 logged in.
>
> The content of vomsdir looks fine to me, but obviously I
> can't see any
> of your more subtle configuration issues (are you using
> Argus?)
>
>
> Trying as ngs.ac.uk <http://ngs.ac.uk>, that doesn't work
> either (and you might as well fix
> it while you are fiddling)
>
> walker@heppc300:~/grid/ngs$ uberftp abaddon.hec.lancs.ac.uk
> <http://abaddon.hec.lancs.ac.uk>
> 220 abaddon.hec.lancs.ac.uk <http://abaddon.hec.lancs.ac.uk>
> GridFTP Server 6.10 (gcc64, 1334324800-83)
> [Globus Toolkit 5.2.0] ready.
> 530-Login incorrect. : globus_gss_assist: Error invoking callout
> 530-globus_callout_module: The callout returned an error
> 530-an unknown error occurred
> 530 End.
> walker@heppc300:~/grid/ngs$ voms-proxy-info --all
> subject :
> /C=UK/O=eScience/OU=__QueenMaryLondon/L=Physics/CN=__christopher
> walker/CN=proxy
> issuer :
> /C=UK/O=eScience/OU=__QueenMaryLondon/L=Physics/CN=__christopher
> walker
> identity :
> /C=UK/O=eScience/OU=__QueenMaryLondon/L=Physics/CN=__christopher
> walker
> type : proxy
> strength : 1024 bits
> path : /tmp/x509up_u32184
> timeleft : 11:31:39
> === VO ngs.ac.uk <http://ngs.ac.uk> extension information ===
> VO : ngs.ac.uk <http://ngs.ac.uk>
> subject :
> /C=UK/O=eScience/OU=__QueenMaryLondon/L=Physics/CN=__christopher
> walker
> issuer :
> /C=UK/O=eScience/OU=__Manchester/L=HEP/CN=voms.__gridpp.ac.uk <http://voms.gridpp.ac.uk>
> attribute : /ngs.ac.uk/Role=NULL/__Capability=NULL
> <http://ngs.ac.uk/Role=NULL/Capability=NULL>
> timeleft : 11:31:39
> uri : voms.gridpp.ac.uk:15010 <http://voms.gridpp.ac.uk:15010>
>
>
> Looking at your that machine:
>
> walker@heppc300:~/grid/ngs$ uberftp abaddon.hec.lancs.ac.uk
> <http://abaddon.hec.lancs.ac.uk>
> 220 abaddon.hec.lancs.ac.uk <http://abaddon.hec.lancs.ac.uk>
> GridFTP Server 6.10 (gcc64, 1334324800-83)
> [Globus Toolkit 5.2.0] ready.
> 230 User dteam116 logged in.
> uberftp> cd /etc/grid-security/vomsdir/ngs__.ac.uk
> <http://ngs.ac.uk>
> uberftp> ls
> -rw-r--r-- 1 root root 64 Sep 13 13:00 15010.lsc
> -rw-r--r-- 1 root root 148 Sep 13 13:25 voms.ngs.ac.uk.lsc
> uberftp> cat 15010.lsc
> ngs.ac.uk <http://ngs.ac.uk>
> /C=UK/O=eScienceCA/OU=__Authority/CN=UK e-Science CA 2B
>
>
> I suspect you are missing a field in your site-info.def (or
> have an
> extra one) for the ngs VO.
>
> No, this doesn't help with t2k.org <http://t2k.org> I'm afraid.
>
> Chris
>
> Cheers,
> Daniela
>
>
> On 19 December 2012 10:47, Matt Doidge
> <[log in to unmask] <mailto:[log in to unmask]>
> <mailto:[log in to unmask]
> <mailto:[log in to unmask]>>> wrote:
>
> Hello all, I hope I caught some of you before you headed
> off for the
> holidays!
>
> Lancaster has been trying to get T2K working on our
> clusters, and on
> our occasionally quirky shared cluster T2K are
> consistently failing
> to successfully submit jobs via the WMS (well
> technically submission
> works, the jobs get aborted), with an incredibly verbose
> error
> message (replicated below, you can also see it in the ticket
> https://ggus.eu/ws/ticket_ info.php?ticket=88628
> <https://ggus.eu/ws/ticket___info.php?ticket=88628
> <https://ggus.eu/ws/ticket_info.php?ticket=88628>>).
>
> The error message looks like either an authentication,
> permissions
> or missing destination problem - but I've checked our CE and
> everything seems okay. As a test I asked Jon to uberftp
> into our CE,
> and he did so without problem as an sgmt2k user.
>
> I'm a little stuck, and would appreciate someone who speaks
> glite-wms-job-status error message to take a look and
> maybe pinpoint
> where in the chain things are breaking. I've learnt the
> hard way
> that the CREAM/WMS interaction is quite complex, and I'm
> wondering
> if this is one of the cases where this has screwed up
> (the two have
> become "out of sync" somehow).
>
> Thanks in advance, and Merry Christmas!
> Matt
>
> ======================= glite-wms-job-status Success
> =====================
> BOOKKEEPING INFORMATION:
>
> Status info for the Job : https://lcglb04.gridpp.rl.ac.
> uk:9000/31MHBsdtFOj7AFEY7rs-lg
> <https://lcglb04.gridpp.rl.ac.__uk:9000/31MHBsdtFOj7AFEY7rs-lg
> <https://lcglb04.gridpp.rl.ac.uk:9000/31MHBsdtFOj7AFEY7rs-lg>__>
> Current Status: Aborted
> Logged Reason(s):
> - Cannot move ISB (retry_copy ${globus_transfer_cmd}
> gsiftp://lcgwms02.gridpp.rl.
> ac.uk:2811/var/SandboxDir/31/
> <http://ac.uk:2811/var/SandboxDir/31/>
> https_3a_2f_2flcglb04.gridpp. rl.ac.uk_3a9000_
> 2f31MHBsdtFOj7AFEY7rs-lg/ input/pexpect.py
> <http://lcgwms02.gridpp.rl.ac.__uk:2811/var/SandboxDir/31/__https_3a_2f_2flcglb04.gridpp.__rl.ac.uk_3a9000___2f31MHBsdtFOj7AFEY7rs-lg/__input/pexpect.py
> <http://lcgwms02.gridpp.rl.ac.uk:2811/var/SandboxDir/31/https_3a_2f_2flcglb04.gridpp.rl.ac.uk_3a9000_2f31MHBsdtFOj7AFEY7rs-lg/input/pexpect.py>>
>
>
> file:///home/grid/sgmt2k005/ home_cream_174147165/
> CREAM174147165/pexpect.py): error: globus_ftp_client:
> the server
> responded with an error500 500-Command failed. :
> globus_l_gfs_file_open failed.500-globus_xio: Unable to
> open file
> /var/SandboxDir/31/https_3a_ 2f_2flcglb04.gridpp.rl.ac.uk_
> 3a9000_2f31MHBsdtFOj7AFEY7rs-
> lg/input/pexpect.py500-globus_ xio:
> System error in open: No such file or
> directory500-globus_xio: A
> system call failed: No such file or directory500 End.;
> reason=1;
> open /home/grid/sgmt2k005/home_
> cream_174147165/.ssh/id_rsa failed:
> No such file or directory. /usr/shared_apps/admin/etc/
> profile.d/keygen2: line 17: /home/grid/sgmt2k005/home_
> cream_174147165/.ssh/ authorized_keys: No such file or
> directory
> chmod: cannot access `/home/grid/sgmt2k005/home_
> cream_174147165/.ssh/ authorized_keys': No such file or
> directory
> /opt/glite/glite/bin/glite-lb- logevent:
> edg_wll_LogEvent*(): LB
> server (bkserver,lbproxy) store protocol error
> (edg_wll_LogEvent():
> LB server (bkserver,lbproxy) store protocol error;;
> Logging library
> ERROR: LB server (bkserver,lbproxy) store protocol error;;
> edg_wll_DoLogEvent(): edg_wll_log_connect error DNS
> resolver error;;
> edg_wll_gss_connect();; GSS Error: Unknown host)
> /opt/glite/glite/bin/glite-lb- logevent:
> edg_wll_LogEvent*(): LB
> server (bkserver,lbproxy) store protocol error
> (edg_wll_LogEvent():
> LB server (bkserver,lbproxy) store protocol error;;
> Logging library
> ERROR: LB server (bkserver,lbproxy) store protocol error;;
> edg_wll_DoLogEvent(): edg_wll_log_connect error DNS
> resolver error;;
> edg_wll_gss_connect();; GSS Error: Unknown host) Cannot
> move ISB
> (retry_copy ${globus_transfer_cmd}
> gsiftp://lcgwms02.gridpp.rl.
> ac.uk:2811/var/SandboxDir/31/
> <http://ac.uk:2811/var/SandboxDir/31/>
> https_3a_2f_2flcglb04.gridpp.
> rl.ac.uk_3a9000_ 2f31MHBsdtFOj7AFEY7rs-lg/ input/pexpect.py
> <http://lcgwms02.gridpp.rl.ac.__uk:2811/var/SandboxDir/31/__https_3a_2f_2flcglb04.gridpp.__rl.ac.uk_3a9000___2f31MHBsdtFOj7AFEY7rs-lg/__input/pexpect.py
> <http://lcgwms02.gridpp.rl.ac.uk:2811/var/SandboxDir/31/https_3a_2f_2flcglb04.gridpp.rl.ac.uk_3a9000_2f31MHBsdtFOj7AFEY7rs-lg/input/pexpect.py>>
>
>
> file:///home/grid/sgmt2k005/ home_cream_174147165/
> CREAM174147165/pexpect.py): error: globus_ftp_client:
> the server
> responded with an error 500 500-Command failed. :
> globus_l_gfs_file_open failed. 500-globus_xio: Unable to
> open file
> /var/SandboxDir/31/https_3a_ 2f_2flcglb04.gridpp.rl.ac.uk_
> 3a9000_2f31MHBsdtFOj7AFEY7rs- lg/input/pexpect.py
> 500-globus_xio:
> System error in open: No such file or directory
> 500-globus_xio: A
> system call failed: No such file or directory 500 End.
> - Transfer to CREAM failed due to exception: CREAM
> Register raised
> std::exception N5glite2ce16cream_client_ api16cream_
> exceptions30JobSubmissionDisab ledExceptionE
> Status Reason: hit job shallow retry count (10)
> Destination: abaddon.hec.lancs.ac.uk:8443/
> <http://abaddon.hec.lancs.ac.uk:8443/> cream-lsf-hex
> <http://abaddon.hec.lancs.ac.__uk:8443/cream-lsf-hex
> <http://abaddon.hec.lancs.ac.uk:8443/cream-lsf-hex>>
> Submitted: Wed Dec 5 15:36:25 2012 GMT
> ==============================
> ==============================
> ==============
>
>
>
>
> --
> Sent from the pit of despair
>
> ------------------------------__-----------------------------
> [log in to unmask]
> <mailto:[log in to unmask]>
> <mailto:[log in to unmask]
> <mailto:[log in to unmask]>>
> HEP Group/Physics Dep
> Imperial College
> Tel: +44-(0)20-75947810 <tel:%2B44-%280%2920-75947810>
> http://www.hep.ph.ic.ac.uk/~__dbauer/
> <http://www.hep.ph.ic.ac.uk/~dbauer/>
>
>
>
>
> --
> Sent from the pit of despair
>
> -----------------------------------------------------------
> [log in to unmask] <mailto:[log in to unmask]>
> HEP Group/Physics Dep
> Imperial College
> Tel: +44-(0)20-75947810
> http://www.hep.ph.ic.ac.uk/~dbauer/
|