Hi Lydia,

You should not need an argument list with tar if you use the -T switch
(for a file list of filenames) and there is no limit to the number of
files you can ask tar to, er, tar. Also files can have spaces in their
names (but not newlines; for newlines you will need to use NUL separator)

(Otherwise the conventional approach is to use xargs but then you have
to call it with tar -u)

Anyway, happy that it's going well. Keep it busy. We should talk about
deleting the obsolete files.

Cheers
--jens


On 12/01/2016 13:40, Lydia Heck wrote:
>
> Hi Jens and Brian,
>
> I have resumed the arriving this morning and the transfers are going
> well.
>
> As I said before I am dealing with a user who has 4.5M files and I am
> going through sets of 10,000 each time. I tried with 100,000 but I hit
> a boundary in that the argument list for tar was too long. So rather
> than trial and error
> I restricted to a maximum of 10,000.
>
> at the ~4.1368M file I will hit the issue with name containing white
> space.
> So this will be with 24 hours or so. Let us see if my script can
> handle that.
>
> Lydia
>
>
>
>
>  On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
>
>> On 12/01/2016 10:34, Lydia Heck wrote:
>>>
>>> Hi Jens,
>>>
>>> my problems were towards the end of the week (Saturday) and in the end
>>> I canceled those jobs as they did not lead to anywhere. So they will
>>> turn up as "cancelled"
>>>
>> OK, so most of the transfers seem to have been ticking along happily
>> when they were killed but I found one which had clearly stalled:
>>
>> https://lcgfts06.gridpp.rl.ac.uk:8449/var/log/fts3/2016-01-09/data.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk/2016-01-09-1215__data.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk__501464354__95f10488-f5ae-47af-b4ec-c0e419467880
>>
>>
>> This is the /only one/ within the past week, all the others were moving
>> data at about 45MB/s.  Here are the relevant parts of the log:
>>
>> Sat Jan  9 12:34:31 2016 INFO     bytes: 6936682496, avg KB/sec:5916,
>> inst KB/sec:256, elapsed:1145
>> Sat Jan  9 12:34:33 2016 INFO     bytes: 6944808960, avg KB/sec:5913,
>> inst KB/sec:4176, elapsed:1147
>> Sat Jan  9 13:47:37 2016 WARNING  Received signal 15 (Terminated)
>> Sat Jan  9 13:47:37 2016 WARNING  TRANSFER
>> 95f10488-f5ae-47af-b4ec-c0e419467880 canceled by the user
>>
>> Curiously the disk server (gdss621) says at that time:
>> [14456] Sat Jan  9 12:34:33 2016 :: Finished transferring
>> "/284707b3-8e49-4895-e053-01b6f6828eaa".
>> [14456] Sat Jan  9 12:34:33 2016 :: Transfer stats:
>> DATE=20160109123433.322795 HOST=gdss621.gridpp.rl.ac.uk
>> PROG=globus-gridftp-server NL.EVNT=FTP_INFO START=20160109121526.124260
>> USER=:globus-mapping: FILE=/284707b3-8e49-4895-e053-01b6f6828eaa
>> BUFFER=131072 BLOCK=262144 NBYTES=6944808960 VOLUME=/ STREAMS=4
>> STRIPES=1 DEST=[129.234.196.65] TYPE=STOR CODE=226
>>
>> That's about 7GB in ~20 mins, or 35MB/s which does not seem
>> unreasonable.
>>
>> Cheers
>> --jens
>>