So I am just about to start to delete 7238028 files from castor form the subdirectories I/Lydia listed before.
I can confirm none of the have "tar" in their filename.
Brian
-----Original Message-----
From: DiRAC Users [mailto:[log in to unmask]] On Behalf Of Jensen, Jens (STFC,RAL,SC)
Sent: 14 January 2016 10:17
To: [log in to unmask]
Subject: Re: Backup proposal -option 3
Great stuff! Now we need to (a) clean up the old stuff (-> Brian), (b) update Document if necessary (Lydia, Jens), (c) get Jon started ASAP (Jon).
Looking at the toplevel directories (ie just under .../cosma5/data) I see about 140TB. Is that what we expect?
Thanks
-j
On 13/01/2016 17:26, Lydia Heck wrote:
>
> Hi Jens,
>
> the last of the tars for the 4.2M file job is being done as I a
> writing this.
>
> no more issues with respect to white space in files.
>
> I will have to re-do two of the archived directories because of the
> white space issue, but then I have completed one who project. There
> are more to go and I would like to start those. But this is the first
> "clean" transfer.
>
> Lydia
>
>
>
>
> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
>
>> Hi Lydia
>>
>> OK, thanks. I will ask Brian to delete the ones we can delete.
>>
>> Cheers
>> --jens
>>
>> On 12/01/2016 14:12, Lydia Heck wrote:
>>>
>>> Hi Jens
>>>
>>> I had this hangup because of previous error messages. I have now
>>> used 100,000. I think that is enough per tar. It just takes too much
>>> time otherwise to create or extract.
>>>
>>> We could delete all of the other directory structures:
>>>
>>> dp010/dc-gord1 dp010/dc-guer1 dp010/dc-hood1 dp010/dc-port2
>>> dp010/dc-kond1 dp010/dc-maso1
>>>
>>> They have now been tar'ed archived.
>>>
>>> But do not delete:
>>> dp010/dc-mejn1
>>> and
>>> jch
>>>
>>> dp010/dc-mejn1 contains tars in directories and jch contains tars
>>> and very large files and I would like to keep those.
>>>
>>>
>>>
>>> then delete Eagle
>>> dp010
>>> dp012
>>> dp023
>>> dp030
>>> dp033
>>>
>>> Once these have been deleted I will start a parallel stream to
>>> attack those again.
>>>
>>> I will also have to rerun some of the dp010 tars where I was caught
>>> out file and directory names with white space.
>>>
>>> Lydia
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
>>>
>>>> Hi Lydia,
>>>>
>>>> You should not need an argument list with tar if you use the -T
>>>> switch (for a file list of filenames) and there is no limit to the
>>>> number of files you can ask tar to, er, tar. Also files can have
>>>> spaces in their names (but not newlines; for newlines you will need
>>>> to use NUL
>>>> separator)
>>>>
>>>> (Otherwise the conventional approach is to use xargs but then you
>>>> have to call it with tar -u)
>>>>
>>>> Anyway, happy that it's going well. Keep it busy. We should talk
>>>> about deleting the obsolete files.
>>>>
>>>> Cheers
>>>> --jens
>>>>
>>>>
>>>> On 12/01/2016 13:40, Lydia Heck wrote:
>>>>>
>>>>> Hi Jens and Brian,
>>>>>
>>>>> I have resumed the arriving this morning and the transfers are
>>>>> going well.
>>>>>
>>>>> As I said before I am dealing with a user who has 4.5M files and I
>>>>> am going through sets of 10,000 each time. I tried with 100,000
>>>>> but I hit a boundary in that the argument list for tar was too
>>>>> long. So rather than trial and error I restricted to a maximum of
>>>>> 10,000.
>>>>>
>>>>> at the ~4.1368M file I will hit the issue with name containing
>>>>> white space.
>>>>> So this will be with 24 hours or so. Let us see if my script can
>>>>> handle that.
>>>>>
>>>>> Lydia
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
>>>>>
>>>>>> On 12/01/2016 10:34, Lydia Heck wrote:
>>>>>>>
>>>>>>> Hi Jens,
>>>>>>>
>>>>>>> my problems were towards the end of the week (Saturday) and in
>>>>>>> the end I canceled those jobs as they did not lead to anywhere.
>>>>>>> So they will turn up as "cancelled"
>>>>>>>
>>>>>> OK, so most of the transfers seem to have been ticking along
>>>>>> happily when they were killed but I found one which had clearly stalled:
>>>>>>
>>>>>> https://lcgfts06.gridpp.rl.ac.uk:8449/var/log/fts3/2016-01-09/dat
>>>>>> a.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk/2016-01-09-1215__dat
>>>>>> a.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk__501464354__95f10488
>>>>>> -f5ae-47af-b4ec-c0e419467880
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> This is the /only one/ within the past week, all the others were
>>>>>> moving data at about 45MB/s. Here are the relevant parts of the
>>>>>> log:
>>>>>>
>>>>>> Sat Jan 9 12:34:31 2016 INFO bytes: 6936682496, avg
>>>>>> KB/sec:5916,
>>>>>> inst KB/sec:256, elapsed:1145
>>>>>> Sat Jan 9 12:34:33 2016 INFO bytes: 6944808960, avg
>>>>>> KB/sec:5913,
>>>>>> inst KB/sec:4176, elapsed:1147
>>>>>> Sat Jan 9 13:47:37 2016 WARNING Received signal 15 (Terminated)
>>>>>> Sat Jan 9 13:47:37 2016 WARNING TRANSFER
>>>>>> 95f10488-f5ae-47af-b4ec-c0e419467880 canceled by the user
>>>>>>
>>>>>> Curiously the disk server (gdss621) says at that time:
>>>>>> [14456] Sat Jan 9 12:34:33 2016 :: Finished transferring
>>>>>> "/284707b3-8e49-4895-e053-01b6f6828eaa".
>>>>>> [14456] Sat Jan 9 12:34:33 2016 :: Transfer stats:
>>>>>> DATE=20160109123433.322795 HOST=gdss621.gridpp.rl.ac.uk
>>>>>> PROG=globus-gridftp-server NL.EVNT=FTP_INFO
>>>>>> START=20160109121526.124260
>>>>>> USER=:globus-mapping: FILE=/284707b3-8e49-4895-e053-01b6f6828eaa
>>>>>> BUFFER=131072 BLOCK=262144 NBYTES=6944808960 VOLUME=/ STREAMS=4
>>>>>> STRIPES=1 DEST=[129.234.196.65] TYPE=STOR CODE=226
>>>>>>
>>>>>> That's about 7GB in ~20 mins, or 35MB/s which does not seem
>>>>>> unreasonable.
>>>>>>
>>>>>> Cheers
>>>>>> --jens
>>>>>>
>>>>
>>
|
