Just in case I have rebooted the system, and so currently I cannot check.
I am going for lunch and then I will have another go.

Lydia


On Tue, 22 Dec 2015, Jensen, Jens (STFC,RAL,SC) wrote:

> Weird! Which version are you using?
>
> We seem to have fts-rest-3.3.3-2 and fts-rest-cli-3.3.3 and
> fts-rest-cloud-storage-3.3.3 and python-fts-3.3.3 but every other fts
> package on the server is 3.3.2. (There is both a python-fts and an
> fts-python - weird).
>
> Cheers
> --jens
>
> On 22/12/2015 12:22, Lydia Heck wrote:
>> Hi Jens,
>>
>> I have a script that I could test. However I now have an issue that the
>>
>> fts-transfer command does not work anymore with the error message
>>
>>
>> fts client is connecting using the gSOAP interface. Consider changing
>>           your configured fts endpoint port to select the REST interface
>>
>> I am currently rebooting the system, but have you seen something
>> similar once before?
>>
>> Best wishes,
>> Lydia
>>
>>
>>
>>
>>
>>
>> On Fri, 18 Dec 2015, Jens Jensen wrote:
>>
>>> Right, so the find suggestion at least would do a depth first listing of
>>> files-to-add, and tar I am guessing would also add files depth first,
>>> which I think meets your requirement, or close enough, of putting
>>> related files into the same chunk.
>>>
>>> Using find-and-then-tar you could avoid building the following archive
>>> until the current one has been sent off to RAL. You'd just need space
>>> for the filelist.
>>>
>>> What I am thinking is:
>>> 1. find <folder to be backed up> -newer <timestamp file> |<list size and
>>> full filename> >filelist
>>> 2. Walk through filelist one line at a time adding up sizes and
>>> filenames till a certain threshold size has been exceeded (say 20GB or
>>> 100,000 files, whichever comes firsts) or adding the next file will take
>>> us above a higher threshold (say 50GB)
>>> 3. Once a list has been found, tar it up, compress it, optionally store
>>> the contents (list) somewhere, send the tarball to RAL, and then
>>> delete it.
>>> 4. Go back to step 2 until the filelist has been completed.
>>> 5. Touch the timestamp file
>>> 6. sleep 24 hours (or whatever) and go to step 1.
>>>
>>> This would meet all our requirements and would be stupidly easy to do.
>>>
>>> Cheers
>>> --jens
>>>
>>>
>>> On 17/12/2015 12:43, Lydia Heck wrote:
>>>>
>>>> Hi Jens,
>>>>
>>>> it took longer than I thought to tidy up the results from the meeting
>>>> last week (I spent a full day on a spreadsheet :-) )
>>>>
>>>> However I am now going to look at the transfers again.
>>>>
>>>> I looked over the presentation you shared with us. And yes, that is
>>>> the way it should go. There are some provisos:
>>>>
>>>> If I create 3 TB chunks, I need to have space for several of them:
>>>>
>>>> One being transfered, one in waiting and one being prepared. This will
>>>> add 10 TB to the storage that is not available for the users; can be
>>>> done, but needs to be factored in.
>>>>
>>>>
>>>> If there is indeed a failure, then I need to identify where the data
>>>> are that have been deleted, corrupted or whatever. If I "just" chunk
>>>> the whole filesystem, that would be difficult, if not impossible to
>>>> find. So I would need to arrange transfers by project, and even then
>>>> the retrieval might physically not be possible, depending of how many
>>>> of the chunks I would have to retrieve.
>>>>
>>>> I believe that currently the biggest top folder is ~500 TB.
>>>>
>>>> There would not be lots of jobs running, simply because there is not
>>>> enough space to chunk that much.
>>>>
>>>> On the storage that I would like to archive there are more than 64M
>>>> files.
>>>>
>>>> So would  a "flat" chunking tar of all the filesystem be a "good"
>>>> idea? I am not sure.
>>>>
>>>> I need to think about this a bit more.
>>>>
>>>> Lydia
>>>>
>>>>
>>>>
>>>>
>>>>  On Thu, 10 Dec 2015, Jensen, Jens (STFC,RAL,SC) wrote:
>>>>
>>>>> Hi Lydia,
>>>>>
>>>>> That's great. I am actually on leave tomorrow (travelling) and out
>>>>> Monday (at Royal Holloway) but the others on the list can follow up.
>>>>>
>>>>> Thanks
>>>>> --jens
>>>>>
>>>>> On 10/12/2015 10:21, Lydia Heck wrote:
>>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> sorry for my silence. I have a meeting in London on Tuesday and
>>>>>> attended CIUK yesterday. Just back and I have to tidy up some
>>>>>> spreadsheets from Tuesday's meeting and I will be busy today as well
>>>>>> with local tasks. So I should get back to this tomorrow.
>>>>>>
>>>>>> Best wishes,
>>>>>> Lydia
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>  On Wed, 9 Dec 2015, Jensen, Jens (STFC,RAL,SC) wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Here is the proposal the third option. Would also be worth looking
>>>>>>> into.
>>>>>>> It is written in python AFAIK.
>>>>>>>
>>>>>>> Overall we are trying to deploy something that meets the
>>>>>>> requirements
>>>>>>> and saves us time in the long run.
>>>>>>>
>>>>>>> Thanks
>>>>>>> --jens
>>>>>>>
>>>>>>>
>>>>>
>>>
>