Btw problem seems to have occured at ~12am ( resolution is bad hence
the ~) Sunday night/ monday morning. is it a weekly cron job gone bad?
There is some acytivity over FTs between MAN and RAL, but that could
well be on the other SE.
Brian
2008/6/17 brian davies <[log in to unmask]>:
> currently getting errorrs between manchester and RAL.
> [FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during
> PREPARATION phase: [GENERAL_FAILURE] Req
> uestFileStatus#-2147069179 failed with error:[ at Tue Jun 17 08:48:03
> BST 2008 state Failed : file not found
> : Problem in get(OSM)StorageInfo : java.io.IOException: No such file
> or directory]] Source Host [dcache01.ti
> er2.hep.manchester.ac.uk] 72
>
> This is one of the error messages for one of these errors:
>
> 2008-06-17 08:37:28,379 [INFO ] - Transfer ID :
> UKINORTHGRIDMANHEP-RALLCG2__2008-06-17-0837_6HjZU6
> 2008-06-17 08:37:28,379 [INFO ] - User DN : xxxxxx
> 2008-06-17 08:37:28,379 [INFO ] - User Descr. : Belonging to FTS job
> [9c3d7cfb-3c48-11dd-a134-9f43a519ff0a]
> 2008-06-17 08:37:28,379 [INFO ] - Source SRM [1.1.0]:
> httpg://dcache01.tier2.hep.manchester.ac.uk:8443/srm/managerv1
> 2008-06-17 08:37:28,379 [INFO ] - Dest. SRM [2.2.0]:
> httpg://srm-atlas.gridpp.rl.ac.uk:8443/srm/managerv2
> 2008-06-17 08:37:28,379 [INFO ] - Source :
> srm://dcache01.tier2.hep.manchester.ac.uk/pnfs/tier2.hep.manchester.ac.uk/data/atlas/valid2/HITS/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.simul.HITS.e113_s417_tid022721/HITS.022721._46413.pool.root.1
> 2008-06-17 08:37:28,379 [INFO ] - Destination:
> srm://srm-atlas.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/atlas/simStrip/atlasmcdisk/valid2/HITS/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.simul.HITS.e113_s417_tid022721_sub01938805/HITS.022721._46413.pool.root.1__DQ2-1213691837
> 2008-06-17 08:37:28,510 [INFO ] - Source SRM server available
> 2008-06-17 08:37:28,730 [INFO ] - Destination SRM server available
> 2008-06-17 08:37:28,730 [INFO ] - STATUS:BEGIN:SOURCE - Preparation
> 2008-06-17 08:37:28,730 [INFO ] - Getting source from SURL
> [srm://dcache01.tier2.hep.manchester.ac.uk/pnfs/tier2.hep.manchester.ac.uk/data/atlas/valid2/HITS/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.simul.HITS.e113_s417_tid022721/HITS.022721._46413.pool.root.1]
> 2008-06-17 08:37:28,895 [INFO ] - PrepareToGet [-2147069106] started
> 2008-06-17 08:37:28,895 [INFO ] - Token : -2147069106
> 2008-06-17 08:37:28,895 [INFO ] - Status : SRM_REQUEST_QUEUED
> 2008-06-17 08:37:28,895 [INFO ] - Message :
> 2008-06-17 08:37:28,895 [INFO ] - > File :
> srm://dcache01.tier2.hep.manchester.ac.uk/pnfs/tier2.hep.manchester.ac.uk/data/atlas/valid2/HITS/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.simul.HITS.e113_s417_tid022721/HITS.022721._46413.pool.root.1
> 2008-06-17 08:37:28,895 [INFO ] - > Status : SRM_REQUEST_QUEUED
> 2008-06-17 08:37:28,895 [INFO ] - > Message :
> 2008-06-17 08:37:28,895 [INFO ] - > Size : 0
> 2008-06-17 08:37:28,895 [INFO ] - > TURL :
> 2008-06-17 08:37:33,043 [INFO ] - Status of PrepareToGet [-2147069106] updated
> 2008-06-17 08:37:33,043 [INFO ] - Token : -2147069106
> 2008-06-17 08:37:33,043 [INFO ] - Status : SRM_INVALID_REQUEST
> 2008-06-17 08:37:33,043 [INFO ] - Message : at Tue Jun 17
> 09:37:28 BST 2008 state Pending : created
> RequestFileStatus#-2147069105 failed with error:[ at Tue Jun 17
> 09:37:29 BST 2008 state Failed : file not found : Problem in
> get(OSM)StorageInfo : java.io.IOException: No such file or directory]
>
> 2008-06-17 08:37:33,043 [INFO ] - > File :
> srm://dcache01.tier2.hep.manchester.ac.uk/pnfs/tier2.hep.manchester.ac.uk/data/atlas/valid2/HITS/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.simul.HITS.e113_s417_tid022721/HITS.022721._46413.pool.root.1
> 2008-06-17 08:37:33,043 [INFO ] - > Status : SRM_FAILURE
> 2008-06-17 08:37:33,043 [INFO ] - > Message :
> RequestFileStatus#-2147069105 failed with error:[ at Tue Jun 17
> 09:37:29 BST 2008 state Failed : file not found : Problem in
> get(OSM)StorageInfo : java.io.IOException: No such file or directory]
> 2008-06-17 08:37:33,043 [INFO ] - > Size : 0
> 2008-06-17 08:37:33,043 [INFO ] - > TURL :
> 2008-06-17 08:37:33,043 [ERROR] - PrepareToGet [-2147069106] failed
> 2008-06-17 08:37:33,043 [ERROR] - source failed during PREPARATION
> phase. Error [GENERAL_FAILURE]:RequestFileStatus#-2147069105 failed
> with error:[ at Tue Jun 17 09:37:29 BST 2008 state Failed : file not
> found : Problem in get(OSM)StorageInfo : java.io.IOException: No such
> file or directory]
> 2008-06-17 08:37:33,043 [INFO ] - STATUS:END fail:SOURCE - Preparation
> 2008-06-17 08:37:33,043 [ERROR] - Final error on SOURCE during
> PREPARATION phase: [GENERAL_FAILURE] RequestFileStatus#-2147069105
> failed with error:[ at Tue Jun 17 09:37:29 BST 2008 state Failed :
> file not found : Problem in get(OSM)StorageInfo : java.io.IOException:
> No such file or directory]
> 2008-06-17 08:37:33,043 [INFO ] - FINAL:SOURCE:
> RequestFileStatus#-2147069105 failed with error:[ at Tue Jun 17
> 09:37:29 BST 2008 state Failed : file not found : Problem in
> get(OSM)StorageInfo : java.io.IOException: No such file or directory]
> 2008-06-17 08:37:33,043 [INFO ] - STATUS:BEGIN:SOURCE - Finalization
> 2008-06-17 08:37:33,043 [INFO ] - Releasing PrepareToGet [-2147069106]
> for SURL [srm://dcache01.tier2.hep.manchester.ac.uk/pnfs/tier2.hep.manchester.ac.uk/data/atlas/valid2/HITS/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.simul.HITS.e113_s417_tid022721/HITS.022721._46413.pool.root.1]
> 2008-06-17 08:37:33,161 [WARN ] - ReleaseFiles for [-2147069106] failed
> 2008-06-17 08:37:33,161 [WARN ] - failed to release PrepareToGet
> [-2147069106]. Try to abort it
> 2008-06-17 08:37:33,161 [INFO ] - Abort completed for request [-2147069106]
> 2008-06-17 08:37:33,161 [INFO ] - PrepareToGet request [-2147069106] aborted
> 2008-06-17 08:37:33,161 [INFO ] - STATUS:END:SOURCE - Finalization
> 2008-06-17 08:37:33,161 [INFO ] - STATUS:BEGIN:DESTINATION - Finalization
> 2008-06-17 08:37:33,161 [INFO ] - No request token provided for
> destination file. Assuming PrepareToPut request has not been sent
> 2008-06-17 08:37:33,161 [INFO ] - STATUS:END:DESTINATION - Finalization
> 2008-06-17 08:37:33,161 [INFO ] - FINAL:fail
>
> Is the file exist
> srm://dcache01.tier2.hep.manchester.ac.uk/pnfs/tier2.hep.manchester.ac.uk/data/atlas/valid2/HITS/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.simul.HITS.e113_s417_tid022721/HITS.022721._46413.pool.root.1
>
> When trying and srmcp -2 -debug=true from dcache@RAL to [log in to unmask] get:
>
> SRMClientV2 : srmCopy, contacting service
> httpg://fal-pygrid-30.lancs.ac.uk:8446/srm/managerv2
> Tue Jun 17 09:49:43 BST 2008: srm returned requestToken = null
> Tue Jun 17 09:49:43 BST 2008: java.io.IOException: srmCopy submission
> failed, unexpected or failed return status : SRM_NOT_SUPPORTED
> explanation=null
> Tue Jun 17 09:49:43 BST 2008: Releasing all remaining file requests
> SRMClientV2 : srmAbortFiles, contacting service
> httpg://fal-pygrid-30.lancs.ac.uk:8446/srm/managerv2
> [main] ERROR ser.BeanSerializer - Exception:
> java.io.IOException: Non nillable element 'requestToken' is null.
> at org.apache.axis.encoding.ser.BeanSerializer.serialize(BeanSerializer.java:215)
> at org.apache.axis.encoding.SerializationContext.serializeActual(SerializationContext.java:1502)
> at org.apache.axis.encoding.SerializationContext.serialize(SerializationContext.java:978)
> at org.apache.axis.encoding.SerializationContext.serialize(SerializationContext.java:799)
> at org.apache.axis.message.RPCParam.serialize(RPCParam.java:208)
> at org.apache.axis.message.RPCElement.outputImpl(RPCElement.java:433)
> at org.apache.axis.message.MessageElement.output(MessageElement.java:1208)
> at org.apache.axis.message.SOAPBody.outputImpl(SOAPBody.java:139)
> at org.apache.axis.message.SOAPEnvelope.outputImpl(SOAPEnvelope.java:478)
> at org.apache.axis.message.MessageElement.output(MessageElement.java:1208)
> at org.apache.axis.SOAPPart.writeTo(SOAPPart.java:315)
> at org.apache.axis.SOAPPart.writeTo(SOAPPart.java:269)
> at org.apache.axis.SOAPPart.saveChanges(SOAPPart.java:530)
> at org.apache.axis.SOAPPart.getContentLength(SOAPPart.java:229)
> at org.apache.axis.Message.getContentLength(Message.java:510)
> at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:371)
> at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)
> at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
> at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
> at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
> at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
> at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
> at org.apache.axis.client.Call.invoke(Call.java:2767)
> at org.apache.axis.client.Call.invoke(Call.java:2443)
> at org.apache.axis.client.Call.invoke(Call.java:2366)
> at org.apache.axis.client.Call.invoke(Call.java:1812)
> at org.dcache.srm.v2_2.SrmSoapBindingStub.srmAbortFiles(SrmSoapBindingStub.java:2523)
> at org.dcache.srm.client.SRMClientV2.srmAbortFiles(SRMClientV2.java:194)
> at gov.fnal.srm.util.SRMCopyClientV2.abortAllPendingFiles(SRMCopyClientV2.java:428)
> at gov.fnal.srm.util.SRMCopyClientV2.start(SRMCopyClientV2.java:388)
> at gov.fnal.srm.util.SRMDispatcher.work(SRMDispatcher.java:779)
> at gov.fnal.srm.util.SRMDispatcher.main(SRMDispatcher.java:372)
> SRMClientV2 : put: try # 0 failed with error
> SRMClientV2 : ; nested exception is:
> java.io.IOException: java.io.IOException: Non nillable element
> 'requestToken' is null.
> SRMClientV2 : put: try again
> SRMClientV2 : sleeping for 10000 milliseconds before retrying
> Tue Jun 17 09:49:43 BST 2008: stopping
> Tue Jun 17 09:49:43 BST 2008: Releasing all remaining file requests
> SRMClientV2 : srmAbortFiles, contacting service
> httpg://fal-pygrid-30.lancs.ac.uk:8446/srm/managerv2
> [Thread-0] ERROR ser.BeanSerializer - Exception:
> java.io.IOException: Non nillable element 'requestToken' is null.
> at org.apache.axis.encoding.ser.BeanSerializer.serialize(BeanSerializer.java:215)
> at org.apache.axis.encoding.SerializationContext.serializeActual(SerializationContext.java:1502)
> at org.apache.axis.encoding.SerializationContext.serialize(SerializationContext.java:978)
>
>
> Does manchester have a srmv2.2 working?
> Brian
>
> A second set of errors is probably just a load/timeout issue
>
> [FTS] FTS State [Failed] FTS Retries [1] Reason [SOURCE error during
> PREPARATION phase: [REQUEST_TIMEOUT] fai
> led to prepare source file in 180 seconds] Source Host
> [dcache01.tier2.hep.manchester.ac.uk] 45
>
> 2008/6/13 Sergey <[log in to unmask]>:
>> Hi Brian,
>>
>> The problem seems been fixed. We passing test.
>> Please try again and let us know about result.
>>
>> Sergey
>>
>> 2008/6/11 brian davies <[log in to unmask]>:
>>> thanks sergey, is therre an eta for a fix?
>>> Brian
>>>
>>> 2008/6/11 Sergey <[log in to unmask]>:
>>>> Hi Brian,
>>>>
>>>> Manchester dcache01 is down now. Since yesterday evening we have a
>>>> problems with upgrade. Hope to sort it out ASAP.
>>>>
>>>> Sergey
>>>>
>>>> 2008/6/11 brian davies <[log in to unmask]>:
>>>>> Has anyone seen a "Protocol(s) specified not supported" error message before?
>>>>> This is on transfers between Manchester and RAL ( errors in both
>>>>> directions both at Manchester
>>>>> a Man-RAL error is shown below.
>>>>> Brian
>>>>>
>>>>>
>>>>> 2008-06-11 07:57:00,792 [INFO ] - Transfer ID :
>>>>> UKINORTHGRIDMANHEP-RALLCG2__2008-06-11-0757_n6lspT
>>>>> 2008-06-11 07:57:00,792 [INFO ] - User DN : xxxxxx
>>>>> 2008-06-11 07:57:00,792 [INFO ] - User Descr. : Belonging to FTS job
>>>>> [f6240371-378b-11dd-bdff-b73269d97bd9]
>>>>> 2008-06-11 07:57:00,792 [INFO ] - Source SRM [1.1.0]:
>>>>> httpg://dcache01.tier2.hep.manchester.ac.uk:8443/srm/managerv1
>>>>> 2008-06-11 07:57:00,792 [INFO ] - Dest. SRM [2.2.0]:
>>>>> httpg://srm-atlas.gridpp.rl.ac.uk:8443/srm/managerv2
>>>>> 2008-06-11 07:57:00,792 [INFO ] - Source :
>>>>> srm://dcache01.tier2.hep.manchester.ac.uk/pnfs/tier2.hep.manchester.ac.uk/data/atlas/valid2/log/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.digit.log.e113_s417_tid022721/log.022721._18053.job.log.tgz.1
>>>>> 2008-06-11 07:57:00,792 [INFO ] - Destination:
>>>>> srm://srm-atlas.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/atlas/simStrip/atlasmcdisk/valid2/log/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.digit.log.e113_s417_tid022721_sub01911950/log.022721._18053.job.log.tgz.1__DQ2-1213171010
>>>>> 2008-06-11 07:57:00,911 [INFO ] - Source SRM server available
>>>>> 2008-06-11 07:57:01,128 [INFO ] - Destination SRM server available
>>>>> 2008-06-11 07:57:01,128 [INFO ] - STATUS:BEGIN:SOURCE - Preparation
>>>>> 2008-06-11 07:57:01,128 [INFO ] - Getting source from SURL
>>>>> [srm://dcache01.tier2.hep.manchester.ac.uk/pnfs/tier2.hep.manchester.ac.uk/data/atlas/valid2/log/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.digit.log.e113_s417_tid022721/log.022721._18053.job.log.tgz.1]
>>>>> 2008-06-11 07:57:01,370 [INFO ] - PrepareToGet [-1] started
>>>>> 2008-06-11 07:57:01,370 [INFO ] - Token : -1
>>>>> 2008-06-11 07:57:01,370 [INFO ] - Status : SRM_FAILURE
>>>>> 2008-06-11 07:57:01,370 [INFO ] - Message : Protocol(s)
>>>>> specified not supported: [ gsiftp ]
>>>>> 2008-06-11 07:57:01,370 [INFO ] - > File :
>>>>> srm://dcache01.tier2.hep.manchester.ac.uk/pnfs/tier2.hep.manchester.ac.uk/data/atlas/valid2/log/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.digit.log.e113_s417_tid022721/log.022721._18053.job.log.tgz.1
>>>>> 2008-06-11 07:57:01,370 [INFO ] - > Status : SRM_FAILURE
>>>>> 2008-06-11 07:57:01,370 [INFO ] - > Message :
>>>>> 2008-06-11 07:57:01,370 [INFO ] - > Size : 0
>>>>> 2008-06-11 07:57:01,370 [INFO ] - > TURL :
>>>>> 2008-06-11 07:57:01,370 [ERROR] - PrepareToGet [-1] failed
>>>>> 2008-06-11 07:57:01,370 [ERROR] - source failed during PREPARATION
>>>>> phase. Error [GENERAL_FAILURE]:source file failed on the SRM with
>>>>> error [SRM_FAILURE]
>>>>> 2008-06-11 07:57:01,370 [INFO ] - STATUS:END fail:SOURCE - Preparation
>>>>> 2008-06-11 07:57:01,370 [ERROR] - Final error on SOURCE during
>>>>> PREPARATION phase: [GENERAL_FAILURE] source file failed on the SRM
>>>>> with error [SRM_FAILURE]
>>>>> 2008-06-11 07:57:01,370 [INFO ] - FINAL:SOURCE: source file failed on
>>>>> the SRM with error [SRM_FAILURE]
>>>>> 2008-06-11 07:57:01,370 [INFO ] - STATUS:BEGIN:SOURCE - Finalization
>>>>> 2008-06-11 07:57:01,370 [INFO ] - Releasing PrepareToGet [-1] for SURL
>>>>> [srm://dcache01.tier2.hep.manchester.ac.uk/pnfs/tier2.hep.manchester.ac.uk/data/atlas/valid2/log/valid2.008801.Hijing_PbPb_5p5TeV_MinBias.digit.log.e113_s417_tid022721/log.022721._18053.job.log.tgz.1]
>>>>> 2008-06-11 07:57:01,740 [WARN ] - ReleaseFiles for [-1] failed
>>>>> 2008-06-11 07:57:01,740 [WARN ] - failed to release PrepareToGet [-1].
>>>>> Try to abort it
>>>>> 2008-06-11 07:57:01,862 [INFO ] - Abort completed for request [-1]
>>>>> 2008-06-11 07:57:01,863 [INFO ] - PrepareToGet request [-1] aborted
>>>>> 2008-06-11 07:57:01,863 [INFO ] - STATUS:END:SOURCE - Finalization
>>>>> 2008-06-11 07:57:01,863 [INFO ] - STATUS:BEGIN:DESTINATION - Finalization
>>>>> 2008-06-11 07:57:01,863 [INFO ] - No request token provided for
>>>>> destination file. Assuming PrepareToPut request has not been sent
>>>>> 2008-06-11 07:57:01,863 [INFO ] - STATUS:END:DESTINATION - Finalization
>>>>> 2008-06-11 07:57:01,863 [INFO ] - FINAL:fail
>>>>>
>>>>
>>>
>>
>
|
