 |
 |
Hi, We are seeing problem with FTS CMS ccrc08 transfers at IC-HEP. Error message: ============================================================== 2008-05-07 14:41:03,571 [INFO ] - Transferred 95682560 out of 2318085925 bytes [4.13%]. Inst: 3104.32 KB/s. Avg: 3104.32 KB/s 2008-05-07 14:46:03,131 [ERROR] - No transfer markers received for more than 300 seconds 2008-05-07 14:46:03,133 [WARN ] - transfer timed out 2008-05-07 14:46:03,134 [WARN ] - gridftp_copy failed: [TRANSFER_TIMEOUT] globus_gass_copy_register_url_to_url: Connection timed out 2008-05-07 14:46:03,134 [INFO ] - STATUS:END fail:TRANSFER - globus_gass_copy_register_url_to_url: Connection timed out ============================================================== A straight srmcp from lx08 seems to work fine. srmcp -2 -debug -streams_num=1 srm://srm-cms.gridpp.rl.ac.uk:8443/srm/managerv2?SFN=/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/BEE7E31A-33A9-DC11-8BB7-0030488A1164.root srm://gfe02.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/BEE7E31A-33A9-DC11-8BB7-0030488A1164.test ============================================================== The fts error message from the log file is as follows: ================================================================== FTS Log File /var/tmp/glite-url-copy-edguser/RALLCG2-UKILT2ICHEPfailed/RALLCG2-UKILT2ICHEP__2008-05-07-1440_a8QqvN.log 2008-05-07 14:40:19,370 [INFO ] - Transfer ID : RALLCG2-UKILT2ICHEP__2008-05-07-1440_a8QqvN 2008-05-07 14:40:19,370 [INFO ] - User DN : xxxxxx 2008-05-07 14:40:19,370 [INFO ] - User Descr. : Belonging to FTS job [81726fda-1c43-11dd-998d-effca8b9cbf5] 2008-05-07 14:40:19,370 [INFO ] - Source SRM [2.2.0]: httpg://srm-cms.gridpp.rl.ac.uk:8443/srm/managerv2 2008-05-07 14:40:19,370 [INFO ] - Dest. SRM [2.2.0]: httpg://gfe02.hep.ph.ic.ac.uk:8443/srm/managerv2 2008-05-07 14:40:19,370 [INFO ] - Source : srm://srm-cms.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root 2008-05-07 14:40:19,370 [INFO ] - Destination: srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root 2008-05-07 14:40:19,587 [INFO ] - Source SRM server available 2008-05-07 14:40:19,685 [INFO ] - Destination SRM server available 2008-05-07 14:40:19,685 [INFO ] - STATUS:BEGIN:SOURCE - Preparation 2008-05-07 14:40:19,685 [INFO ] - Getting source from SURL [srm://srm-cms.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root] 2008-05-07 14:40:20,761 [INFO ] - PrepareToGet [3928469] started 2008-05-07 14:40:20,761 [INFO ] - Token : 3928469 2008-05-07 14:40:20,762 [INFO ] - Status : SRM_REQUEST_QUEUED 2008-05-07 14:40:20,762 [INFO ] - Message : 2008-05-07 14:40:20,762 [INFO ] - > File : srm://srm-cms.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root 2008-05-07 14:40:20,762 [INFO ] - > Status : SRM_REQUEST_QUEUED 2008-05-07 14:40:20,762 [INFO ] - > Message : 2008-05-07 14:40:20,762 [INFO ] - > Size : 2318085925 2008-05-07 14:40:20,762 [INFO ] - > TURL : 2008-05-07 14:40:26,344 [INFO ] - Status of PrepareToGet [3928469] updated 2008-05-07 14:40:26,344 [INFO ] - Token : 3928469 2008-05-07 14:40:26,344 [INFO ] - Status : SRM_SUCCESS 2008-05-07 14:40:26,344 [INFO ] - Message : 2008-05-07 14:40:26,344 [INFO ] - > File : srm://srm-cms.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root 2008-05-07 14:40:26,344 [INFO ] - > Status : SRM_FILE_PINNED 2008-05-07 14:40:26,344 [INFO ] - > Message : 2008-05-07 14:40:26,344 [INFO ] - > Size : 2318085925 2008-05-07 14:40:26,344 [INFO ] - > TURL : gsiftp://gdss85.gridpp.rl.ac.uk:2811//castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root 2008-05-07 14:40:26,344 [INFO ] - PrepareToGet [3928469] completed 2008-05-07 14:40:26,344 [INFO ] - STATUS:END:SOURCE - Preparation 2008-05-07 14:40:26,344 [INFO ] - STATUS:BEGIN:DESTINATION - Preparation 2008-05-07 14:40:26,344 [INFO ] - Getting destination from SURL [srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root] 2008-05-07 14:40:29,764 [INFO ] - path [srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root] does not exist on SRM [httpg://gfe02.hep.ph.ic.ac.uk:8443/srm/managerv2] 2008-05-07 14:40:29,890 [INFO ] - PrepareToPut [-2146212457] started 2008-05-07 14:40:29,890 [INFO ] - Token : -2146212457 2008-05-07 14:40:29,890 [INFO ] - Space Token : 2008-05-07 14:40:29,891 [INFO ] - Status : SRM_REQUEST_QUEUED 2008-05-07 14:40:29,891 [INFO ] - Message : at Wed May 07 15:40:29 BST 2008 state Pending : created 2008-05-07 14:40:29,891 [INFO ] - > File : srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root 2008-05-07 14:40:29,891 [INFO ] - > Status : SRM_REQUEST_QUEUED 2008-05-07 14:40:29,891 [INFO ] - > Message : at Wed May 07 15:40:29 BST 2008 state TQueued : put on the thread queue 2008-05-07 14:40:29,891 [INFO ] - > Size : 2318085925 2008-05-07 14:40:29,891 [INFO ] - > TURL : 2008-05-07 14:40:31,006 [INFO ] - Status of PrepareToPut [-2146212457] updated 2008-05-07 14:40:31,006 [INFO ] - Token : -2146212457 2008-05-07 14:40:31,006 [INFO ] - Space Token : 2008-05-07 14:40:31,006 [INFO ] - Status : SRM_SUCCESS 2008-05-07 14:40:31,006 [INFO ] - Message : at Wed May 07 15:40:29 BST 2008 state Pending : created 2008-05-07 14:40:31,006 [INFO ] - > File : srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root 2008-05-07 14:40:31,006 [INFO ] - > Status : SRM_SPACE_AVAILABLE 2008-05-07 14:40:31,006 [INFO ] - > Message : at Wed May 07 15:40:30 BST 2008 state Ready : execution succeeded 2008-05-07 14:40:31,006 [INFO ] - > Size : 2318085925 2008-05-07 14:40:31,006 [INFO ] - > TURL : gsiftp://sedsk01.hep.ph.ic.ac.uk:2811///store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root 2008-05-07 14:40:31,006 [INFO ] - PrepareToPut [-2146212457] completed 2008-05-07 14:40:31,006 [INFO ] - STATUS:END:DESTINATION - Preparation 2008-05-07 14:40:31,524 [INFO ] - source verification completed 2008-05-07 14:40:31,524 [INFO ] - STATUS:BEGIN:TRANSFER 2008-05-07 14:40:31,524 [INFO ] - starting gsiftp transfer 2008-05-07 14:40:31,524 [INFO ] - GSIFTP: source: set up FTP mode. DCAU disabled. Streams = 1, Tcp BS = 0 2008-05-07 14:40:31,524 [INFO ] - GSIFTP: dest: set up FTP mode. DCAU disabled. Streams = 1, Tcp BS = 0 2008-05-07 14:41:03,571 [INFO ] - Transferred 95682560 out of 2318085925 bytes [4.13%]. Inst: 3104.32 KB/s. Avg: 3104.32 KB/s 2008-05-07 14:46:03,131 [ERROR] - No transfer markers received for more than 300 seconds 2008-05-07 14:46:03,133 [WARN ] - transfer timed out 2008-05-07 14:46:03,134 [WARN ] - gridftp_copy failed: [TRANSFER_TIMEOUT] globus_gass_copy_register_url_to_url: Connection timed out 2008-05-07 14:46:03,134 [INFO ] - STATUS:END fail:TRANSFER - globus_gass_copy_register_url_to_url: Connection timed out 2008-05-07 14:46:03,134 [WARN ] - gsiftp transfer failed 2008-05-07 14:46:03,134 [ERROR] - transfer failed during TRANSFER phase. Error [TRANSFER_TIMEOUT]:globus_gass_copy_register_url_to_url: Connection timed out 2008-05-07 14:46:03,134 [ERROR] - Final error on TRANSFER during TRANSFER phase: [TRANSFER_TIMEOUT] globus_gass_copy_register_url_to_url: Connection timed out 2008-05-07 14:46:03,134 [INFO ] - FINAL:TRANSFER: globus_gass_copy_register_url_to_url: Connection timed out 2008-05-07 14:46:03,134 [ERROR] - transfer failed 2008-05-07 14:46:03,134 [INFO ] - STATUS:BEGIN:SOURCE - Finalization 2008-05-07 14:46:03,134 [INFO ] - Releasing PrepareToGet [3928469] for SURL [srm://srm-cms.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root] 2008-05-07 14:46:03,389 [INFO ] - PrepareToGet [3928469] successfully released 2008-05-07 14:46:03,389 [INFO ] - STATUS:END:SOURCE - Finalization 2008-05-07 14:46:03,389 [INFO ] - STATUS:BEGIN:DESTINATION - Finalization 2008-05-07 14:46:03,389 [INFO ] - aborting PrepareToPut [-2146212457] for SURL [srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root] 2008-05-07 14:46:03,608 [INFO ] - Abort completed for request [-2146212457] 2008-05-07 14:46:03,608 [INFO ] - PrepareToPut [-2146212457] successfully aborted 2008-05-07 14:46:03,608 [ERROR] - destination failed during FINALIZATION phase. Error [GENERAL_FAILURE]:Request aborted 2008-05-07 14:46:04,042 [INFO ] - File [srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root] does no longer exist 2008-05-07 14:46:04,042 [INFO ] - STATUS:END fail:DESTINATION - Finalization 2008-05-07 14:46:04,042 [INFO ] - FINAL:fail =================================================================== Has anyone seen this before and any idea how can we solve it, is it possible to get help from developers as it is very high priority issue for CMS. On our dcache we have setup performanceMarkerPeriod=30 Thanks, Mona -- Mona Aggarwal- Imperial College Tel: +442075947809 Email: [log in to unmask]