Hi Maarten,

the command
grep hosts /etc/nsswitch.conf
gives
hosts:      nis files dns

but the file /etc/nsswitch.conf has not been changed since 21st april.
I don't know if there has been changed something on the WN's since I 
have no root access there.

Here ist the output of the globus-url-copy command with the -dbg option:

-bash-2.05b$ globus-url-copy -dbg file:/etc/group 
gsiftp://ekp-lcg-ce.physik.uni-karlsruhe.de/tmp/test.$$
debug: starting to put 
gsiftp://ekp-lcg-ce.physik.uni-karlsruhe.de/tmp/test.31247
debug: connecting to 
gsiftp://ekp-lcg-ce.physik.uni-karlsruhe.de/tmp/test.31247
debug: response from 
gsiftp://ekp-lcg-ce.physik.uni-karlsruhe.de/tmp/test.31247:
220 ekp-lcg-ce.physik.uni-karlsruhe.de GridFTP Server 1.12 GSSAPI type 
Globus/GSI wu-2.6.2 (gcc32dbg, 1062606889-42) ready.

debug: authenticating with 
gsiftp://ekp-lcg-ce.physik.uni-karlsruhe.de/tmp/test.31247
debug: error reading response from 
gsiftp://ekp-lcg-ce.physik.uni-karlsruhe.de/tmp/test.31247: 
globus_l_ftp_control_send_cmd_cb: gss_init_sec_context failed

 GSS failure:
GSS Major Status: Authentication Failed
GSS Minor Status Error Chain:

init_sec_context.c:171: gss_init_sec_context: SSLv3 handshake problems
globus_i_gsi_gss_utils.c:881: globus_i_gsi_gss_handshake: Unable to 
verify remote side's credentials
globus_i_gsi_gss_utils.c:854: globus_i_gsi_gss_handshake: SSLv3 
handshake problems: Couldn't do ssl handshake
OpenSSL Error: s3_clnt.c:840: in library: SSL routines, function 
SSL3_GET_SERVER_CERTIFICATE: certificate verify failed
globus_gsi_callback.c:351: globus_i_gsi_callback_handshake_callback: 
Could not verify credential
globus_gsi_callback.c:477: globus_i_gsi_callback_cred_verify: Could not 
verify credential
globus_gsi_callback.c:769: globus_i_gsi_callback_check_revoked: Invalid 
CRL: The available CRL has expired
debug: fault on connection to 
gsiftp://ekp-lcg-ce.physik.uni-karlsruhe.de/tmp/test.31247: 
globus_l_ftp_control_send_cmd_cb: gss_init_sec_context failed

 GSS failure:
GSS Major Status: Authentication Failed
GSS Minor Status Error Chain:

init_sec_context.c:171: gss_init_sec_context: SSLv3 handshake problems
globus_i_gsi_gss_utils.c:881: globus_i_gsi_gss_handshake: Unable to 
verify remote side's credentials
globus_i_gsi_gss_utils.c:854: globus_i_gsi_gss_handshake: SSLv3 
handshake problems: Couldn't do ssl handshake
OpenSSL Error: s3_clnt.c:840: in library: SSL routines, function 
SSL3_GET_SERVER_CERTIFICATE: certificate verify failed
globus_gsi_callback.c:351: globus_i_gsi_callback_handshake_callback: 
Could not verify credential
globus_gsi_callback.c:477: globus_i_gsi_callback_cred_verify: Could not 
verify credential
globus_gsi_callback.c:769: globus_i_gsi_callback_check_revoked: Invalid 
CRL: The available CRL has expired
debug: data callback, error globus_l_ftp_control_send_cmd_cb: 
gss_init_sec_context failed

 GSS failure:
GSS Major Status: Authentication Failed
GSS Minor Status Error Chain:

init_sec_context.c:171: gss_init_sec_context: SSLv3 handshake problems
globus_i_gsi_gss_utils.c:881: globus_i_gsi_gss_handshake: Unable to 
verify remote side's credentials
globus_i_gsi_gss_utils.c:854: globus_i_gsi_gss_handshake: SSLv3 
handshake problems: Couldn't do ssl handshake
OpenSSL Error: s3_clnt.c:840: in library: SSL routines, function 
SSL3_GET_SERVER_CERTIFICATE: certificate verify failed
globus_gsi_callback.c:351: globus_i_gsi_callback_handshake_callback: 
Could not verify credential
globus_gsi_callback.c:477: globus_i_gsi_callback_cred_verify: Could not 
verify credential
globus_gsi_callback.c:769: globus_i_gsi_callback_check_revoked: Invalid 
CRL: The available CRL has expired, buffer 0xb7111008, length 0, 
offset=0, eof=true
debug: operation complete
error: globus_l_ftp_control_send_cmd_cb: gss_init_sec_context failed

 GSS failure:
GSS Major Status: Authentication Failed
GSS Minor Status Error Chain:

init_sec_context.c:171: gss_init_sec_context: SSLv3 handshake problems
globus_i_gsi_gss_utils.c:881: globus_i_gsi_gss_handshake: Unable to 
verify remote side's credentials
globus_i_gsi_gss_utils.c:854: globus_i_gsi_gss_handshake: SSLv3 
handshake problems: Couldn't do ssl handshake
OpenSSL Error: s3_clnt.c:840: in library: SSL routines, function 
SSL3_GET_SERVER_CERTIFICATE: certificate verify failed
globus_gsi_callback.c:351: globus_i_gsi_callback_handshake_callback: 
Could not verify credential
globus_gsi_callback.c:477: globus_i_gsi_callback_cred_verify: Could not 
verify credential
globus_gsi_callback.c:769: globus_i_gsi_callback_check_revoked: Invalid 
CRL: The available CRL has expired


Thank you for your support,
Anja



Maarten Litmaath wrote:

> Anja Vest wrote:
>
>> Hi Maarten,
>> last week our job submission problem was gone after we set in 
>> /etc/hosts :
>>
>> ----------------------------------------------------------------------------- 
>>
>> 192.168.101.231 ekp-lcg-ce.physik.uni-karlsruhe.de     
>> ekp-lcg-ce.ekpplus.cluster     ekp-lcg-ce
>> ----------------------------------------------------------------------------- 
>>
>>
>> as you proposed.
>> According to your reply to GGUS ticket#2492, the WN's now resolve the 
>> SE ekp-lcg-se.physik.uni-karlsruhe.de
>> to its public address 129.13.133.13
>> The entries in /etc/hosts now look like this:
>>
>> # LCG stuff
>> 192.168.101.220 ekp-lcg-ui.physik.uni-karlsruhe.de      
>> ekp-lcg-ui      ekp-lcg-ui.ekpplus.cluster
>> 192.168.101.231 ekp-lcg-ce.physik.uni-karlsruhe.de      
>> ekp-lcg-ce      ekp-lcg-ce.ekpplus.cluster
>> 192.168.101.232 ekp-lcg-se.ekpplus.cluster
>> 129.13.133.13 ekp-lcg-se.physik.uni-karlsruhe.de      ekp-lcg-se
>> 129.13.133.14   ekp-lcg-mon.physik.uni-karlsruhe.de     ekp-lcg-mon
>>
>> This has been working since sunday morning (or saturday afternoon).
>>  From then on a grid job shows the same symptoms as before.
>> Trying to do the test in
>> http://goc.grid.sinica.edu.tw/gocwiki/submit-helper_script_%2e%2e%2e_gave_error%3a_cache_export_dir_%2e%2e%2e 
>>
>> and following all the diagnosis steps, I just got:
>>
>> ekpplus021:~>globus-url-copy file:/etc/group 
>> gsiftp://ekp-lcg-ce.physik.uni-karlsruhe.de/tmp/test.$$
>> error: globus_l_ftp_control_send_cmd_cb: gss_init_sec_context failed
>>
>> GSS failure:
>> GSS Major Status: Authentication Failed
>> GSS Minor Status Error Chain:
>> ...
>
>
> Please rerun that command with the "-dbg" option: maybe this time the 
> problem
> is due to something else.  Did you change more things on the WN?  For 
> example,
> what does this command report:
>
>     grep hosts /etc/nsswitch.conf
>
>> Did you do the test via a grid job ? (I think I have to do so as well 
>> since I have no root access on our WN's)
>
>
> I submitted a job to the jobmanager-fork on your CE, that submitted a 
> test job
> to your batch system with qsub.
>
>> Could it be possible, that also the CE needs to be resolved to its 
>> public address on the WN's?
>
>
> If the original fix worked last week, we should be able to get it to 
> work again
> without resorting to using the public address, which was in fact 
> proposed as an
> alternative solution.
>
>> cheers,
>> Anja
>