HI Brian,
sorry there is more:
there are subdirectories in dp010 that can/should be deleted:
dc-gord1
dc-guer1
dc-hood1
dc-hugh1
dc-jone1
dc-kond1
dc-maso1
dc-ohar1
dc-port2
dc-stur1
But definitely NOT dc-mejn1
I am very sorry, this is just tedious and we should have done this before I
started the new lot. But nothing is really lost even if there is a mistake.
There is now a good route to get the archiving done.
Best wishes,
Lydia
On Thu, 14 Jan 2016, Lydia Heck wrote:
>
> Hi Brian,
>
> yes please could you delete:
>
> The directories and contents
>
> Eagle
> dp012
> dp023
> dp030
> dp033
>
> DO NOT delete dp010!!!!
> and NOT
> dp010/dc-mejn1 and jch
>
> my list previously contained dp010 - but that was a HUGE mistake.
>
> Lydia
>
>
>
>
>
>
>>>>
>>>> They have now been tar'ed archived.
>>>>
>>>> But do not delete:
>>>> dp010/dc-mejn1
>>>> and
>>>> jch
>>>>
>>>> dp010/dc-mejn1 contains tars in directories and jch contains tars
>>>> and very large files and I would like to keep those.
>>>>
>>>>
>>>>
>>>> then delete Eagle
>>>> dp010
>>>> dp012
>>>> dp023
>
> On Thu, 14 Jan 2016, Brian Davies wrote:
>
>> So I am just about to start to delete 7238028 files from castor form the
>> subdirectories I/Lydia listed before.
>> I can confirm none of the have "tar" in their filename.
>> Brian
>>
>> -----Original Message-----
>> From: DiRAC Users [mailto:[log in to unmask]] On Behalf Of Jensen,
>> Jens (STFC,RAL,SC)
>> Sent: 14 January 2016 10:17
>> To: [log in to unmask]
>> Subject: Re: Backup proposal -option 3
>>
>> Great stuff! Now we need to (a) clean up the old stuff (-> Brian), (b)
>> update Document if necessary (Lydia, Jens), (c) get Jon started ASAP (Jon).
>>
>> Looking at the toplevel directories (ie just under .../cosma5/data) I see
>> about 140TB. Is that what we expect?
>>
>> Thanks
>> -j
>>
>> On 13/01/2016 17:26, Lydia Heck wrote:
>>>
>>> Hi Jens,
>>>
>>> the last of the tars for the 4.2M file job is being done as I a
>>> writing this.
>>>
>>> no more issues with respect to white space in files.
>>>
>>> I will have to re-do two of the archived directories because of the
>>> white space issue, but then I have completed one who project. There
>>> are more to go and I would like to start those. But this is the first
>>> "clean" transfer.
>>>
>>> Lydia
>>>
>>>
>>>
>>>
>>> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
>>>
>>>> Hi Lydia
>>>>
>>>> OK, thanks. I will ask Brian to delete the ones we can delete.
>>>>
>>>> Cheers
>>>> --jens
>>>>
>>>> On 12/01/2016 14:12, Lydia Heck wrote:
>>>>>
>>>>> Hi Jens
>>>>>
>>>>> I had this hangup because of previous error messages. I have now
>>>>> used 100,000. I think that is enough per tar. It just takes too much
>>>>> time otherwise to create or extract.
>>>>>
>>>>> We could delete all of the other directory structures:
>>>>>
>>>>> dp010/dc-gord1 dp010/dc-guer1 dp010/dc-hood1 dp010/dc-port2
>>>>> dp010/dc-kond1 dp010/dc-maso1
>>>>>
>>>>> They have now been tar'ed archived.
>>>>>
>>>>> But do not delete:
>>>>> dp010/dc-mejn1
>>>>> and
>>>>> jch
>>>>>
>>>>> dp010/dc-mejn1 contains tars in directories and jch contains tars
>>>>> and very large files and I would like to keep those.
>>>>>
>>>>>
>>>>>
>>>>> then delete Eagle
>>>>> dp010
>>>>> dp012
>>>>> dp023
>>>>> dp030
>>>>> dp033
>>>>>
>>>>> Once these have been deleted I will start a parallel stream to
>>>>> attack those again.
>>>>>
>>>>> I will also have to rerun some of the dp010 tars where I was caught
>>>>> out file and directory names with white space.
>>>>>
>>>>> Lydia
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
>>>>>
>>>>>> Hi Lydia,
>>>>>>
>>>>>> You should not need an argument list with tar if you use the -T
>>>>>> switch (for a file list of filenames) and there is no limit to the
>>>>>> number of files you can ask tar to, er, tar. Also files can have
>>>>>> spaces in their names (but not newlines; for newlines you will need
>>>>>> to use NUL
>>>>>> separator)
>>>>>>
>>>>>> (Otherwise the conventional approach is to use xargs but then you
>>>>>> have to call it with tar -u)
>>>>>>
>>>>>> Anyway, happy that it's going well. Keep it busy. We should talk
>>>>>> about deleting the obsolete files.
>>>>>>
>>>>>> Cheers
>>>>>> --jens
>>>>>>
>>>>>>
>>>>>> On 12/01/2016 13:40, Lydia Heck wrote:
>>>>>>>
>>>>>>> Hi Jens and Brian,
>>>>>>>
>>>>>>> I have resumed the arriving this morning and the transfers are
>>>>>>> going well.
>>>>>>>
>>>>>>> As I said before I am dealing with a user who has 4.5M files and I
>>>>>>> am going through sets of 10,000 each time. I tried with 100,000
>>>>>>> but I hit a boundary in that the argument list for tar was too
>>>>>>> long. So rather than trial and error I restricted to a maximum of
>>>>>>> 10,000.
>>>>>>>
>>>>>>> at the ~4.1368M file I will hit the issue with name containing
>>>>>>> white space.
>>>>>>> So this will be with 24 hours or so. Let us see if my script can
>>>>>>> handle that.
>>>>>>>
>>>>>>> Lydia
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote:
>>>>>>>
>>>>>>>> On 12/01/2016 10:34, Lydia Heck wrote:
>>>>>>>>>
>>>>>>>>> Hi Jens,
>>>>>>>>>
>>>>>>>>> my problems were towards the end of the week (Saturday) and in
>>>>>>>>> the end I canceled those jobs as they did not lead to anywhere.
>>>>>>>>> So they will turn up as "cancelled"
>>>>>>>>>
>>>>>>>> OK, so most of the transfers seem to have been ticking along
>>>>>>>> happily when they were killed but I found one which had clearly
>>>>>>>> stalled:
>>>>>>>>
>>>>>>>> https://lcgfts06.gridpp.rl.ac.uk:8449/var/log/fts3/2016-01-09/dat
>>>>>>>> a.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk/2016-01-09-1215__dat
>>>>>>>> a.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk__501464354__95f10488
>>>>>>>> -f5ae-47af-b4ec-c0e419467880
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This is the /only one/ within the past week, all the others were
>>>>>>>> moving data at about 45MB/s. Here are the relevant parts of the
>>>>>>>> log:
>>>>>>>>
>>>>>>>> Sat Jan 9 12:34:31 2016 INFO bytes: 6936682496, avg
>>>>>>>> KB/sec:5916,
>>>>>>>> inst KB/sec:256, elapsed:1145
>>>>>>>> Sat Jan 9 12:34:33 2016 INFO bytes: 6944808960, avg
>>>>>>>> KB/sec:5913,
>>>>>>>> inst KB/sec:4176, elapsed:1147
>>>>>>>> Sat Jan 9 13:47:37 2016 WARNING Received signal 15 (Terminated)
>>>>>>>> Sat Jan 9 13:47:37 2016 WARNING TRANSFER
>>>>>>>> 95f10488-f5ae-47af-b4ec-c0e419467880 canceled by the user
>>>>>>>>
>>>>>>>> Curiously the disk server (gdss621) says at that time:
>>>>>>>> [14456] Sat Jan 9 12:34:33 2016 :: Finished transferring
>>>>>>>> "/284707b3-8e49-4895-e053-01b6f6828eaa".
>>>>>>>> [14456] Sat Jan 9 12:34:33 2016 :: Transfer stats:
>>>>>>>> DATE=20160109123433.322795 HOST=gdss621.gridpp.rl.ac.uk
>>>>>>>> PROG=globus-gridftp-server NL.EVNT=FTP_INFO
>>>>>>>> START=20160109121526.124260
>>>>>>>> USER=:globus-mapping: FILE=/284707b3-8e49-4895-e053-01b6f6828eaa
>>>>>>>> BUFFER=131072 BLOCK=262144 NBYTES=6944808960 VOLUME=/ STREAMS=4
>>>>>>>> STRIPES=1 DEST=[129.234.196.65] TYPE=STOR CODE=226
>>>>>>>>
>>>>>>>> That's about 7GB in ~20 mins, or 35MB/s which does not seem
>>>>>>>> unreasonable.
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> --jens
>>>>>>>>
>>>>>>
>>>>
>>
>
|