Great stuff! Now we need to (a) clean up the old stuff (-> Brian), (b)
update Document if necessary (Lydia, Jens), (c) get Jon started ASAP (Jon).
Looking at the toplevel directories (ie just under .../cosma5/data) I
see about 140TB. Is that what we expect?
Thanks
-j
On 13/01/2016 17:26, Lydia Heck wrote:
>
> Hi Jens,
>
> the last of the tars for the 4.2M file job is being done as I a
> writing this.
>
> no more issues with respect to white space in files.
>
> I will have to re-do two of the archived directories because of the
> white space issue, but then I have completed one who project. There
> are more to go and I would like to start those. But this is the first
> "clean" transfer.
>
> Lydia
>
>
>
>
> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
>
>> Hi Lydia
>>
>> OK, thanks. I will ask Brian to delete the ones we can delete.
>>
>> Cheers
>> --jens
>>
>> On 12/01/2016 14:12, Lydia Heck wrote:
>>>
>>> Hi Jens
>>>
>>> I had this hangup because of previous error messages. I have now used
>>> 100,000. I think that is enough per tar. It just takes too much time
>>> otherwise to create or
>>> extract.
>>>
>>> We could delete all of the other directory structures:
>>>
>>> dp010/dc-gord1 dp010/dc-guer1 dp010/dc-hood1 dp010/dc-port2
>>> dp010/dc-kond1 dp010/dc-maso1
>>>
>>> They have now been tar'ed archived.
>>>
>>> But do not delete:
>>> dp010/dc-mejn1
>>> and
>>> jch
>>>
>>> dp010/dc-mejn1 contains tars in directories
>>> and
>>> jch contains tars and very large files and I would like to keep those.
>>>
>>>
>>>
>>> then delete Eagle
>>> dp010
>>> dp012
>>> dp023
>>> dp030
>>> dp033
>>>
>>> Once these have been deleted I will start a parallel stream to attack
>>> those again.
>>>
>>> I will also have to rerun some of the dp010 tars where I was caught
>>> out file and directory names with white space.
>>>
>>> Lydia
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
>>>
>>>> Hi Lydia,
>>>>
>>>> You should not need an argument list with tar if you use the -T switch
>>>> (for a file list of filenames) and there is no limit to the number of
>>>> files you can ask tar to, er, tar. Also files can have spaces in their
>>>> names (but not newlines; for newlines you will need to use NUL
>>>> separator)
>>>>
>>>> (Otherwise the conventional approach is to use xargs but then you have
>>>> to call it with tar -u)
>>>>
>>>> Anyway, happy that it's going well. Keep it busy. We should talk about
>>>> deleting the obsolete files.
>>>>
>>>> Cheers
>>>> --jens
>>>>
>>>>
>>>> On 12/01/2016 13:40, Lydia Heck wrote:
>>>>>
>>>>> Hi Jens and Brian,
>>>>>
>>>>> I have resumed the arriving this morning and the transfers are going
>>>>> well.
>>>>>
>>>>> As I said before I am dealing with a user who has 4.5M files and I am
>>>>> going through sets of 10,000 each time. I tried with 100,000 but I
>>>>> hit
>>>>> a boundary in that the argument list for tar was too long. So rather
>>>>> than trial and error
>>>>> I restricted to a maximum of 10,000.
>>>>>
>>>>> at the ~4.1368M file I will hit the issue with name containing white
>>>>> space.
>>>>> So this will be with 24 hours or so. Let us see if my script can
>>>>> handle that.
>>>>>
>>>>> Lydia
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
>>>>>
>>>>>> On 12/01/2016 10:34, Lydia Heck wrote:
>>>>>>>
>>>>>>> Hi Jens,
>>>>>>>
>>>>>>> my problems were towards the end of the week (Saturday) and in the
>>>>>>> end
>>>>>>> I canceled those jobs as they did not lead to anywhere. So they
>>>>>>> will
>>>>>>> turn up as "cancelled"
>>>>>>>
>>>>>> OK, so most of the transfers seem to have been ticking along happily
>>>>>> when they were killed but I found one which had clearly stalled:
>>>>>>
>>>>>> https://lcgfts06.gridpp.rl.ac.uk:8449/var/log/fts3/2016-01-09/data.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk/2016-01-09-1215__data.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk__501464354__95f10488-f5ae-47af-b4ec-c0e419467880
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> This is the /only one/ within the past week, all the others were
>>>>>> moving
>>>>>> data at about 45MB/s. Here are the relevant parts of the log:
>>>>>>
>>>>>> Sat Jan 9 12:34:31 2016 INFO bytes: 6936682496, avg
>>>>>> KB/sec:5916,
>>>>>> inst KB/sec:256, elapsed:1145
>>>>>> Sat Jan 9 12:34:33 2016 INFO bytes: 6944808960, avg
>>>>>> KB/sec:5913,
>>>>>> inst KB/sec:4176, elapsed:1147
>>>>>> Sat Jan 9 13:47:37 2016 WARNING Received signal 15 (Terminated)
>>>>>> Sat Jan 9 13:47:37 2016 WARNING TRANSFER
>>>>>> 95f10488-f5ae-47af-b4ec-c0e419467880 canceled by the user
>>>>>>
>>>>>> Curiously the disk server (gdss621) says at that time:
>>>>>> [14456] Sat Jan 9 12:34:33 2016 :: Finished transferring
>>>>>> "/284707b3-8e49-4895-e053-01b6f6828eaa".
>>>>>> [14456] Sat Jan 9 12:34:33 2016 :: Transfer stats:
>>>>>> DATE=20160109123433.322795 HOST=gdss621.gridpp.rl.ac.uk
>>>>>> PROG=globus-gridftp-server NL.EVNT=FTP_INFO
>>>>>> START=20160109121526.124260
>>>>>> USER=:globus-mapping: FILE=/284707b3-8e49-4895-e053-01b6f6828eaa
>>>>>> BUFFER=131072 BLOCK=262144 NBYTES=6944808960 VOLUME=/ STREAMS=4
>>>>>> STRIPES=1 DEST=[129.234.196.65] TYPE=STOR CODE=226
>>>>>>
>>>>>> That's about 7GB in ~20 mins, or 35MB/s which does not seem
>>>>>> unreasonable.
>>>>>>
>>>>>> Cheers
>>>>>> --jens
>>>>>>
>>>>
>>
|