FYI, this problem was eventually understood (more info in the attachment)
Cheers, Massimo
On Wed, 23 Feb 2011, Arnau Bria wrote:
> Hi all,
>
> we're failing some CE SAM tests because of the above message.
>
> log says:
> STATUS CHANGED: PENDING => IDLE [localUser=ops002]
> [gridJobId=https://wms01.egee.cesga.es:9000/NeJF1qazTfd04M1vPSRhqA] [delegationId=12982415712E525280wms012Eegee2Ecesga2Ees] 23 Feb 2011 11:33:17,470 INFO org.glite.ce.creamapi.jobmanagement.cmdexecutor.AbstractJobExecutor (AbstractJobExecutor.java:2163) - (Worker Thread 24) JOB CREAM791346294
>
> STATUS CHANGED: IDLE => RUNNING [localUser=ops002] [gridJobId=https://wms01.egee.cesga.es:9000/NeJF1qazTfd04M1vPSRhqA] [delegationId=12982415712E525280wms012Eegee2Ecesga2Ees]
> 23 Feb 2011 11:33:19,401 INFO org.glite.ce.creamapi.jobmanagement.cmdexecutor.AbstractJobExecutor (AbstractJobExecutor.java:2163) - (Worker Thread 20) JOB CREAM791346294 STATUS CHANGED: RUNNING => DONE-FAILED [failureReason=Cannot move ISB (retry_copy ${globus_transfer_cmd} gsiftp://wms01.egee.cesga.es:2811/var/glite/SandboxDir/Ne/https_3a_2f_2fwms01.egee.cesga.es_3a9000_2fNeJF1qazTfd04M1vPSRhqA/input/nagrun.sh file:///home/ops002/home_cream_791346294/CREAM791346294/nagrun.sh): proxy expired] [localUser=ops002] [gridJobId=https://wms01.egee.cesga.es:9000/NeJF1qazTfd04M1vPSRhqA] [workerNode=td074.pic.es] [delegationId=12982415712E525280wms012Eegee2Ecesga2Ees]
>
>
> I'm looking for some doc about this error but I didn't find any.
> http://grid.pd.infn.it/cream/field.php?n=Main.ErrorMessagesReportedByCREAMToClient
> http://grid.pd.infn.it/cream/field.php?n=Main.KnownIssues
>
> The StandardError of this job:
>
> # cat /opt/glite/var/cream_sandbox/ops/_DC_es_DC_irisgrid_O_cesga_CN_javier_lopez_ops_Role_lcgadmin_Capability_NULL_ops002/79/CREAM791346294/StandardError
> SetLoggingJob(https://wms01.egee.cesga.es:9000/NeJF1qazTfd04M1vPSRhqA,UI=000000:NS=0000000004:WM=000009:BH=0000000000:JSS=000004:LM=000012:LRMS=000000:APP=000000:LBS=000000): GSSAPI Error (failed to load GSI credentials: GSS Major Status: General failure
> (GSS Minor Status Error Chain:
> globus_gsi_gssapi: Error with GSI credential
> globus_gsi_gssapi: Error with gss credential handle
> globus_credential: Error with credential: The proxy credential: /home/ops002/home_cream_791346294/cream_791346294.proxy
> with subject: /DC=es/DC=irisgrid/O=cesga/CN=javier-lopez/CN=proxy/CN=proxy/CN=proxy/CN=proxy/CN=proxy/CN=proxy/CN=limited proxy
> expired 1083 minutes ago.
>
> ))
> SetLoggingJob(https://wms01.egee.cesga.es:9000/NeJF1qazTfd04M1vPSRhqA,UI=000000:NS=0000000004:WM=000009:BH=0000000000:JSS=000004:LM=000012:LRMS=000000:APP=000000:LBS=000000): GSSAPI Error (failed to load GSI credentials: GSS Major Status: General failure
> (GSS Minor Status Error Chain:
> globus_gsi_gssapi: Error with GSI credential
> globus_gsi_gssapi: Error with gss credential handle
> globus_credential: Error with credential: The proxy credential: /home/ops002/home_cream_791346294/cream_791346294.proxy
> with subject: /DC=es/DC=irisgrid/O=cesga/CN=javier-lopez/CN=proxy/CN=proxy/CN=proxy/CN=proxy/CN=proxy/CN=proxy/CN=limited proxy
> expired 1083 minutes ago.
>
> ))
> Cannot move ISB (retry_copy ${globus_transfer_cmd} gsiftp://wms01.egee.cesga.es:2811/var/glite/SandboxDir/Ne/https_3a_2f_2fwms01.egee.cesga.es_3a9000_2fNeJF1qazTfd04M1vPSRhqA/input/nagrun.sh file:///home/ops002/home_cream_791346294/CREAM791346294/nagrun.sh): proxy expired
>
> and querying mysql:
>
> mysql> select dn,local_user,start_time,termination_time,last_update_time,dlg_id from t_credential where dlg_id like '%12982415712E525280%';
> +--------------------------------------------+------------+---------------------+---------------------+---------------------+------------------------------------------+
> | dn | local_user | start_time | termination_time | last_update_time | dlg_id |
> +--------------------------------------------+------------+---------------------+---------------------+---------------------+------------------------------------------+
> | /DC=es/DC=irisgrid/O=cesga/CN=javier-lopez | ops002 | 2011-02-23 07:30:06 | 2011-02-23 17:28:06 | 2011-02-23 07:35:56 | 12982415712E525280wms012Eegee2Ecesga2Ees |
> +--------------------------------------------+------------+---------------------+---------------------+---------------------+------------------------------------------+
>
>
>
> I can submit a job using my proxy and works fine, so the 'service' is
> fine.
>
> $ glite-ce-job-status https://ce08.pic.es:8443/CREAM104266933
> 2011-02-23 11:42:02,951 WARN - No configuration file suitable for loading. Using built-in configuration
>
> ****** JobID=[https://ce08.pic.es:8443/CREAM104266933]
> Status = [DONE-OK]
> ExitCode = [0]
>
> is ops proxy really expired? Anyone suffered from same error and could
> give us some clue?
>
>
> TIA,
> Arnau
>
\|||/
-----------0oo----( o o )----oo0-------------------
(_)
INFN Sezione di Padova
Via Marzolo, 8
35131 Padova - Italy E-mail: massimo.sgaravatto [at] pd.infn.it
Tel: ++39 0498275908 Skype: massimo.sgaravatto
Fax: ++39 0498275952
From [log in to unmask] Wed Feb 23 16:51:37 2011
Date: Wed, 23 Feb 2011 16:51:34 +0100
From: Luigi Zangrando <[log in to unmask]>
To: Massimo Sgaravatto - INFN Padova <[log in to unmask]>
Cc: Daniela Bauer <[log in to unmask]>, Arnau Bria <[log in to unmask]>, [log in to unmask]
Subject: Re: [cream-support] Re: [LCG-ROLLOUT] CREAM CE: Cannot move ISB (retry_copy ${globus_transfer_cmd} ... proxy expired
[ The following text is in the "UTF-8" character set. ]
[ Your display is set for the "ISO-8859-15" character set. ]
[ Some characters may be displayed incorrectly. ]
On Wed, 2011-02-23 at 16:25 +0100, Massimo Sgaravatto - INFN Padova
wrote:
> The problem at PIC is that the renewal was done correctly by the client
> (WMS). The proxy renewal command operation is registered in the database
> but for some reasons (being debugged by the developers) the relevant proxy
> file was not updated.
I understood the problem.
The CREAM-CE exposes two different services: the one for the job
management and the other one for the delegation management. As you can
see from the log files, the cream service failed the startup several
times because of a misconfiguration problem (I believe the cream's
database url was wrong). Meanwhile the delegation service, correctly
configured was up and running. So, in this scenario, a client is able to
create delegations, but is not able to submit jobs.
The delegation 12982415712E525280wms012Eegee2Ecesga2Ees was created
correctly but, the cream service (which was not runnning), was not able
to create the delegation proxy file in the filesystem needed for the
submission to the LRMS.
The day 23 Feb 2011 at 09:10:38 the cream service was restarted
successfully and the 12982415712E525280wms012Eegee2Ecesga2Ees was even
valid! So cream accepted the request for registering the job
CREAM796315178 but the relative proxy file was not in the sandbox. This
is the reason of the failure.
cheers,
L.
|
