JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for GRIDPP-STORAGE Archives


GRIDPP-STORAGE Archives

GRIDPP-STORAGE Archives


GRIDPP-STORAGE@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

GRIDPP-STORAGE Home

GRIDPP-STORAGE Home

GRIDPP-STORAGE  May 2008

GRIDPP-STORAGE May 2008

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Major problem at IC-HEP with FTS CMS ccrc08 transfers

From:

Mona Aggarwal <[log in to unmask]>

Reply-To:

[log in to unmask]

Date:

Thu, 8 May 2008 14:45:28 +0100

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (177 lines)

Hi,

We are seeing problem with FTS CMS ccrc08 transfers at
IC-HEP.

Error message:

==============================================================

2008-05-07 14:41:03,571 [INFO ] - Transferred 95682560 out of
2318085925 bytes [4.13%]. Inst: 3104.32 KB/s.  Avg: 3104.32 KB/s
2008-05-07 14:46:03,131 [ERROR] - No transfer markers received for
more than 300 seconds
2008-05-07 14:46:03,133 [WARN ] - transfer timed out
2008-05-07 14:46:03,134 [WARN ] - gridftp_copy failed:
[TRANSFER_TIMEOUT] globus_gass_copy_register_url_to_url: Connection
timed out
2008-05-07 14:46:03,134 [INFO ] - STATUS:END fail:TRANSFER -
globus_gass_copy_register_url_to_url: Connection timed out

==============================================================
A straight srmcp from lx08 seems to work fine.
srmcp -2 -debug -streams_num=1
srm://srm-cms.gridpp.rl.ac.uk:8443/srm/managerv2?SFN=/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/BEE7E31A-33A9-DC11-8BB7-0030488A1164.root
srm://gfe02.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/BEE7E31A-33A9-DC11-8BB7-0030488A1164.test

==============================================================

The fts error message from the log file is as follows:


==================================================================
FTS Log File
/var/tmp/glite-url-copy-edguser/RALLCG2-UKILT2ICHEPfailed/RALLCG2-UKILT2ICHEP__2008-05-07-1440_a8QqvN.log

2008-05-07 14:40:19,370 [INFO ] - Transfer ID  :
RALLCG2-UKILT2ICHEP__2008-05-07-1440_a8QqvN
2008-05-07 14:40:19,370 [INFO ] - User DN      : xxxxxx
2008-05-07 14:40:19,370 [INFO ] - User Descr.  : Belonging to FTS job
[81726fda-1c43-11dd-998d-effca8b9cbf5]
2008-05-07 14:40:19,370 [INFO ] - Source SRM [2.2.0]:
httpg://srm-cms.gridpp.rl.ac.uk:8443/srm/managerv2
2008-05-07 14:40:19,370 [INFO ] - Dest.  SRM [2.2.0]:
httpg://gfe02.hep.ph.ic.ac.uk:8443/srm/managerv2
2008-05-07 14:40:19,370 [INFO ] - Source     :
srm://srm-cms.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root
2008-05-07 14:40:19,370 [INFO ] - Destination:
srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root
2008-05-07 14:40:19,587 [INFO ] - Source SRM server available
2008-05-07 14:40:19,685 [INFO ] - Destination SRM server available
2008-05-07 14:40:19,685 [INFO ] - STATUS:BEGIN:SOURCE - Preparation
2008-05-07 14:40:19,685 [INFO ] - Getting source from SURL
[srm://srm-cms.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root]
2008-05-07 14:40:20,761 [INFO ] - PrepareToGet [3928469] started
2008-05-07 14:40:20,761 [INFO ] - Token            : 3928469
2008-05-07 14:40:20,762 [INFO ] - Status           : SRM_REQUEST_QUEUED
2008-05-07 14:40:20,762 [INFO ] - Message          :
2008-05-07 14:40:20,762 [INFO ] - > File          :
srm://srm-cms.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root
2008-05-07 14:40:20,762 [INFO ] - > Status        : SRM_REQUEST_QUEUED
2008-05-07 14:40:20,762 [INFO ] - > Message       :
2008-05-07 14:40:20,762 [INFO ] - > Size          : 2318085925
2008-05-07 14:40:20,762 [INFO ] - > TURL          :
2008-05-07 14:40:26,344 [INFO ] - Status of PrepareToGet [3928469] updated
2008-05-07 14:40:26,344 [INFO ] - Token            : 3928469
2008-05-07 14:40:26,344 [INFO ] - Status           : SRM_SUCCESS
2008-05-07 14:40:26,344 [INFO ] - Message          :
2008-05-07 14:40:26,344 [INFO ] - > File          :
srm://srm-cms.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root
2008-05-07 14:40:26,344 [INFO ] - > Status        : SRM_FILE_PINNED
2008-05-07 14:40:26,344 [INFO ] - > Message       :
2008-05-07 14:40:26,344 [INFO ] - > Size          : 2318085925
2008-05-07 14:40:26,344 [INFO ] - > TURL          :
gsiftp://gdss85.gridpp.rl.ac.uk:2811//castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root
2008-05-07 14:40:26,344 [INFO ] - PrepareToGet [3928469] completed
2008-05-07 14:40:26,344 [INFO ] - STATUS:END:SOURCE - Preparation
2008-05-07 14:40:26,344 [INFO ] - STATUS:BEGIN:DESTINATION - Preparation
2008-05-07 14:40:26,344 [INFO ] - Getting destination from SURL
[srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root]
2008-05-07 14:40:29,764 [INFO ] - path
[srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root]
does not exist on SRM
[httpg://gfe02.hep.ph.ic.ac.uk:8443/srm/managerv2]
2008-05-07 14:40:29,890 [INFO ] - PrepareToPut [-2146212457] started
2008-05-07 14:40:29,890 [INFO ] - Token           : -2146212457
2008-05-07 14:40:29,890 [INFO ] - Space Token     :
2008-05-07 14:40:29,891 [INFO ] - Status          : SRM_REQUEST_QUEUED
2008-05-07 14:40:29,891 [INFO ] - Message         :  at Wed May 07
15:40:29 BST 2008 state Pending : created
2008-05-07 14:40:29,891 [INFO ] - > File          :
srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root
2008-05-07 14:40:29,891 [INFO ] - > Status        : SRM_REQUEST_QUEUED
2008-05-07 14:40:29,891 [INFO ] - > Message       :  at Wed May 07
15:40:29 BST 2008 state TQueued : put on the thread queue
2008-05-07 14:40:29,891 [INFO ] - > Size          : 2318085925
2008-05-07 14:40:29,891 [INFO ] - > TURL          :
2008-05-07 14:40:31,006 [INFO ] - Status of PrepareToPut [-2146212457]
updated
2008-05-07 14:40:31,006 [INFO ] - Token           : -2146212457
2008-05-07 14:40:31,006 [INFO ] - Space Token     :
2008-05-07 14:40:31,006 [INFO ] - Status          : SRM_SUCCESS
2008-05-07 14:40:31,006 [INFO ] - Message         :  at Wed May 07
15:40:29 BST 2008 state Pending : created
2008-05-07 14:40:31,006 [INFO ] - > File          :
srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root
2008-05-07 14:40:31,006 [INFO ] - > Status        : SRM_SPACE_AVAILABLE
2008-05-07 14:40:31,006 [INFO ] - > Message       :  at Wed May 07
15:40:30 BST 2008 state Ready : execution succeeded
2008-05-07 14:40:31,006 [INFO ] - > Size          : 2318085925
2008-05-07 14:40:31,006 [INFO ] - > TURL          :
gsiftp://sedsk01.hep.ph.ic.ac.uk:2811///store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root
2008-05-07 14:40:31,006 [INFO ] - PrepareToPut [-2146212457] completed
2008-05-07 14:40:31,006 [INFO ] - STATUS:END:DESTINATION - Preparation
2008-05-07 14:40:31,524 [INFO ] - source verification completed
2008-05-07 14:40:31,524 [INFO ] - STATUS:BEGIN:TRANSFER
2008-05-07 14:40:31,524 [INFO ] - starting gsiftp transfer
2008-05-07 14:40:31,524 [INFO ] - GSIFTP: source: set up FTP mode.
DCAU disabled. Streams = 1, Tcp BS = 0
2008-05-07 14:40:31,524 [INFO ] - GSIFTP: dest: set up FTP mode. DCAU
disabled. Streams = 1, Tcp BS = 0
2008-05-07 14:41:03,571 [INFO ] - Transferred 95682560 out of
2318085925 bytes [4.13%]. Inst: 3104.32 KB/s.  Avg: 3104.32 KB/s
2008-05-07 14:46:03,131 [ERROR] - No transfer markers received for
more than 300 seconds
2008-05-07 14:46:03,133 [WARN ] - transfer timed out
2008-05-07 14:46:03,134 [WARN ] - gridftp_copy failed:
[TRANSFER_TIMEOUT] globus_gass_copy_register_url_to_url: Connection
timed out
2008-05-07 14:46:03,134 [INFO ] - STATUS:END fail:TRANSFER -
globus_gass_copy_register_url_to_url: Connection timed out
2008-05-07 14:46:03,134 [WARN ] - gsiftp transfer failed
2008-05-07 14:46:03,134 [ERROR] - transfer failed during TRANSFER
phase. Error [TRANSFER_TIMEOUT]:globus_gass_copy_register_url_to_url:
Connection timed out
2008-05-07 14:46:03,134 [ERROR] - Final error on TRANSFER during
TRANSFER phase: [TRANSFER_TIMEOUT]
globus_gass_copy_register_url_to_url: Connection timed out
2008-05-07 14:46:03,134 [INFO ] - FINAL:TRANSFER:
globus_gass_copy_register_url_to_url: Connection timed out
2008-05-07 14:46:03,134 [ERROR] - transfer failed
2008-05-07 14:46:03,134 [INFO ] - STATUS:BEGIN:SOURCE - Finalization
2008-05-07 14:46:03,134 [INFO ] - Releasing PrepareToGet [3928469] for
SURL
[srm://srm-cms.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/prod/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root]
2008-05-07 14:46:03,389 [INFO ] - PrepareToGet [3928469] successfully
released
2008-05-07 14:46:03,389 [INFO ] - STATUS:END:SOURCE - Finalization
2008-05-07 14:46:03,389 [INFO ] - STATUS:BEGIN:DESTINATION - Finalization
2008-05-07 14:46:03,389 [INFO ] - aborting PrepareToPut [-2146212457]
for SURL
[srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root]
2008-05-07 14:46:03,608 [INFO ] - Abort completed for request [-2146212457]
2008-05-07 14:46:03,608 [INFO ] - PrepareToPut [-2146212457]
successfully aborted
2008-05-07 14:46:03,608 [ERROR] - destination failed during
FINALIZATION phase. Error [GENERAL_FAILURE]:Request aborted
2008-05-07 14:46:04,042 [INFO ] - File
[srm://gfe02.hep.ph.ic.ac.uk/pnfs/hep.ph.ic.ac.uk/data/cms/store/mc/2007/12/7/CSA07-Wenu-1197047869/0008/103D264F-CEA9-DC11-B7A8-00E081348D01.root]
does no longer exist
2008-05-07 14:46:04,042 [INFO ] - STATUS:END fail:DESTINATION - Finalization
2008-05-07 14:46:04,042 [INFO ] - FINAL:fail
===================================================================

Has anyone seen this before and any idea how can we solve it,
is it possible to get help from developers as it is very high
priority issue for CMS.

On our dcache we have setup performanceMarkerPeriod=30

Thanks,

Mona

-- 
Mona Aggarwal- Imperial College
Tel: +442075947809
Email: [log in to unmask]

Top of Message | Previous Page | Permalink

JISCMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
December 2005
November 2005
October 2005
September 2005
August 2005
July 2005
June 2005
May 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004
September 2004
August 2004
July 2004
June 2004


WWW.JISCMAIL.AC.UK

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager