Hi Jens and Brian,
I have resumed the arriving this morning and the transfers are going well.
As I said before I am dealing with a user who has 4.5M files and I am going
through sets of 10,000 each time. I tried with 100,000 but I hit a boundary in
that the argument list for tar was too long. So rather than trial and error
I restricted to a maximum of 10,000.
at the ~4.1368M file I will hit the issue with name containing white space.
So this will be with 24 hours or so. Let us see if my script can handle that.
Lydia
On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
> On 12/01/2016 10:34, Lydia Heck wrote:
>>
>> Hi Jens,
>>
>> my problems were towards the end of the week (Saturday) and in the end
>> I canceled those jobs as they did not lead to anywhere. So they will
>> turn up as "cancelled"
>>
> OK, so most of the transfers seem to have been ticking along happily
> when they were killed but I found one which had clearly stalled:
>
> https://lcgfts06.gridpp.rl.ac.uk:8449/var/log/fts3/2016-01-09/data.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk/2016-01-09-1215__data.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk__501464354__95f10488-f5ae-47af-b4ec-c0e419467880
>
> This is the /only one/ within the past week, all the others were moving
> data at about 45MB/s. Here are the relevant parts of the log:
>
> Sat Jan 9 12:34:31 2016 INFO bytes: 6936682496, avg KB/sec:5916, inst KB/sec:256, elapsed:1145
> Sat Jan 9 12:34:33 2016 INFO bytes: 6944808960, avg KB/sec:5913, inst KB/sec:4176, elapsed:1147
> Sat Jan 9 13:47:37 2016 WARNING Received signal 15 (Terminated)
> Sat Jan 9 13:47:37 2016 WARNING TRANSFER 95f10488-f5ae-47af-b4ec-c0e419467880 canceled by the user
>
> Curiously the disk server (gdss621) says at that time:
> [14456] Sat Jan 9 12:34:33 2016 :: Finished transferring
> "/284707b3-8e49-4895-e053-01b6f6828eaa".
> [14456] Sat Jan 9 12:34:33 2016 :: Transfer stats:
> DATE=20160109123433.322795 HOST=gdss621.gridpp.rl.ac.uk
> PROG=globus-gridftp-server NL.EVNT=FTP_INFO START=20160109121526.124260
> USER=:globus-mapping: FILE=/284707b3-8e49-4895-e053-01b6f6828eaa
> BUFFER=131072 BLOCK=262144 NBYTES=6944808960 VOLUME=/ STREAMS=4
> STRIPES=1 DEST=[129.234.196.65] TYPE=STOR CODE=226
>
> That's about 7GB in ~20 mins, or 35MB/s which does not seem unreasonable.
>
> Cheers
> --jens
>
|