 |
 |
Hi Brian, yes please could you delete: The directories and contents Eagle dp012 dp023 dp030 dp033 DO NOT delete dp010!!!! and NOT dp010/dc-mejn1 and jch my list previously contained dp010 - but that was a HUGE mistake. Lydia >>> >>> They have now been tar'ed archived. >>> >>> But do not delete: >>> dp010/dc-mejn1 >>> and >>> jch >>> >>> dp010/dc-mejn1 contains tars in directories and jch contains tars >>> and very large files and I would like to keep those. >>> >>> >>> >>> then delete Eagle >>> dp010 >>> dp012 >>> dp023 On Thu, 14 Jan 2016, Brian Davies wrote: > So I am just about to start to delete 7238028 files from castor form the subdirectories I/Lydia listed before. > I can confirm none of the have "tar" in their filename. > Brian > > -----Original Message----- > From: DiRAC Users [mailto:[log in to unmask]] On Behalf Of Jensen, Jens (STFC,RAL,SC) > Sent: 14 January 2016 10:17 > To: [log in to unmask] > Subject: Re: Backup proposal -option 3 > > Great stuff! Now we need to (a) clean up the old stuff (-> Brian), (b) update Document if necessary (Lydia, Jens), (c) get Jon started ASAP (Jon). > > Looking at the toplevel directories (ie just under .../cosma5/data) I see about 140TB. Is that what we expect? > > Thanks > -j > > On 13/01/2016 17:26, Lydia Heck wrote: >> >> Hi Jens, >> >> the last of the tars for the 4.2M file job is being done as I a >> writing this. >> >> no more issues with respect to white space in files. >> >> I will have to re-do two of the archived directories because of the >> white space issue, but then I have completed one who project. There >> are more to go and I would like to start those. But this is the first >> "clean" transfer. >> >> Lydia >> >> >> >> >> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote: >> >>> Hi Lydia >>> >>> OK, thanks. I will ask Brian to delete the ones we can delete. >>> >>> Cheers >>> --jens >>> >>> On 12/01/2016 14:12, Lydia Heck wrote: >>>> >>>> Hi Jens >>>> >>>> I had this hangup because of previous error messages. I have now >>>> used 100,000. I think that is enough per tar. It just takes too much >>>> time otherwise to create or extract. >>>> >>>> We could delete all of the other directory structures: >>>> >>>> dp010/dc-gord1 dp010/dc-guer1 dp010/dc-hood1 dp010/dc-port2 >>>> dp010/dc-kond1 dp010/dc-maso1 >>>> >>>> They have now been tar'ed archived. >>>> >>>> But do not delete: >>>> dp010/dc-mejn1 >>>> and >>>> jch >>>> >>>> dp010/dc-mejn1 contains tars in directories and jch contains tars >>>> and very large files and I would like to keep those. >>>> >>>> >>>> >>>> then delete Eagle >>>> dp010 >>>> dp012 >>>> dp023 >>>> dp030 >>>> dp033 >>>> >>>> Once these have been deleted I will start a parallel stream to >>>> attack those again. >>>> >>>> I will also have to rerun some of the dp010 tars where I was caught >>>> out file and directory names with white space. >>>> >>>> Lydia >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote: >>>> >>>>> Hi Lydia, >>>>> >>>>> You should not need an argument list with tar if you use the -T >>>>> switch (for a file list of filenames) and there is no limit to the >>>>> number of files you can ask tar to, er, tar. Also files can have >>>>> spaces in their names (but not newlines; for newlines you will need >>>>> to use NUL >>>>> separator) >>>>> >>>>> (Otherwise the conventional approach is to use xargs but then you >>>>> have to call it with tar -u) >>>>> >>>>> Anyway, happy that it's going well. Keep it busy. We should talk >>>>> about deleting the obsolete files. >>>>> >>>>> Cheers >>>>> --jens >>>>> >>>>> >>>>> On 12/01/2016 13:40, Lydia Heck wrote: >>>>>> >>>>>> Hi Jens and Brian, >>>>>> >>>>>> I have resumed the arriving this morning and the transfers are >>>>>> going well. >>>>>> >>>>>> As I said before I am dealing with a user who has 4.5M files and I >>>>>> am going through sets of 10,000 each time. I tried with 100,000 >>>>>> but I hit a boundary in that the argument list for tar was too >>>>>> long. So rather than trial and error I restricted to a maximum of >>>>>> 10,000. >>>>>> >>>>>> at the ~4.1368M file I will hit the issue with name containing >>>>>> white space. >>>>>> So this will be with 24 hours or so. Let us see if my script can >>>>>> handle that. >>>>>> >>>>>> Lydia >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, 12 Jan 2016, Jensen, Jens (STFC,RAL,SC) wrote: >>>>>> >>>>>>> On 12/01/2016 10:34, Lydia Heck wrote: >>>>>>>> >>>>>>>> Hi Jens, >>>>>>>> >>>>>>>> my problems were towards the end of the week (Saturday) and in >>>>>>>> the end I canceled those jobs as they did not lead to anywhere. >>>>>>>> So they will turn up as "cancelled" >>>>>>>> >>>>>>> OK, so most of the transfers seem to have been ticking along >>>>>>> happily when they were killed but I found one which had clearly stalled: >>>>>>> >>>>>>> https://lcgfts06.gridpp.rl.ac.uk:8449/var/log/fts3/2016-01-09/dat >>>>>>> a.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk/2016-01-09-1215__dat >>>>>>> a.cosma.dur.ac.uk__srm-dirac.gridpp.rl.ac.uk__501464354__95f10488 >>>>>>> -f5ae-47af-b4ec-c0e419467880 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> This is the /only one/ within the past week, all the others were >>>>>>> moving data at about 45MB/s. Here are the relevant parts of the >>>>>>> log: >>>>>>> >>>>>>> Sat Jan 9 12:34:31 2016 INFO bytes: 6936682496, avg >>>>>>> KB/sec:5916, >>>>>>> inst KB/sec:256, elapsed:1145 >>>>>>> Sat Jan 9 12:34:33 2016 INFO bytes: 6944808960, avg >>>>>>> KB/sec:5913, >>>>>>> inst KB/sec:4176, elapsed:1147 >>>>>>> Sat Jan 9 13:47:37 2016 WARNING Received signal 15 (Terminated) >>>>>>> Sat Jan 9 13:47:37 2016 WARNING TRANSFER >>>>>>> 95f10488-f5ae-47af-b4ec-c0e419467880 canceled by the user >>>>>>> >>>>>>> Curiously the disk server (gdss621) says at that time: >>>>>>> [14456] Sat Jan 9 12:34:33 2016 :: Finished transferring >>>>>>> "/284707b3-8e49-4895-e053-01b6f6828eaa". >>>>>>> [14456] Sat Jan 9 12:34:33 2016 :: Transfer stats: >>>>>>> DATE=20160109123433.322795 HOST=gdss621.gridpp.rl.ac.uk >>>>>>> PROG=globus-gridftp-server NL.EVNT=FTP_INFO >>>>>>> START=20160109121526.124260 >>>>>>> USER=:globus-mapping: FILE=/284707b3-8e49-4895-e053-01b6f6828eaa >>>>>>> BUFFER=131072 BLOCK=262144 NBYTES=6944808960 VOLUME=/ STREAMS=4 >>>>>>> STRIPES=1 DEST=[129.234.196.65] TYPE=STOR CODE=226 >>>>>>> >>>>>>> That's about 7GB in ~20 mins, or 35MB/s which does not seem >>>>>>> unreasonable. >>>>>>> >>>>>>> Cheers >>>>>>> --jens >>>>>>> >>>>> >>> >