On 17/12/13 21:26, Jonathan Perkin wrote:
> Hi Chris,
>
> did anything ever come of this  (i.e. generic script to parse syncat dumps for non LHC VOs)?
>
> One of the reasons I ask is that we’re pretty much at quota at QM and presently I’m not sure why...
>

I dropped the ball on this before Christmas.

Perhaps we should invite Jon to a storage meeting - and see how we can 
help him.

T2k has  270115 files at QMUL.

I have generated a dump using agedu [1] - at:

srm://se03.esc.qmul.ac.uk/t2k.org/agedu.dat
srm://se03.esc.qmul.ac.uk/t2k.org/agedu.dat.dump


which Jon may be able to use to work out where the bulk of the data is.

Here's what I've done to take a look:

  walker@heplt019:~$ cat agedu.dat.dump |  ./pub/src/agedu-r9480/agedu -L
Built pathname index, 270114 entries, 23482781 bytes of index 

Faking directory atimes
Building index
Final index file size = 40518536 bytes
walker@heplt019:~$ ./pub/src/agedu-r9480/agedu -w
Using Linux /proc/net magic authentication
URL: http://localhost:46299/

Chris

[1] http://www.chiark.greenend.org.uk/~sgtatham/agedu/
> Cheers
>
> Jon
>
> On 14 Feb 2013, at 16:45, Christopher J. Walker <[log in to unmask]> wrote:
>
>> David,
>>       This is the other project I was thinking that Janusz should do.
>> Modify the ATLAS LFC<->SE consistency tools to work with non LHC VOs.
>>
>> Again, it has applicability for all non LHC VOs.
>>
>>
>> On 14/02/13 15:40, Jonathan Perkin wrote:
>>>
>>> On 14/02/13 13:00, Christopher J. Walker wrote:
>>>> Jon,
>>>>     I'm about to see if I can get time allocated for someone (probably
>>>> Janucz) to do the following task. I just thought I'd check that this was
>>>> something you'd find useful. I know you have some tools to do this already.
>>>>
>>>> There's a problem with catalogue synchronisation between the LFC and
>>>> what's really on SEs - both files that are in the LFC and not on sites,
>>>> and vice versa. It would be very useful to have some generic scripts
>>>> that solve this problem - specifically for T2K.org, but they would be
>>>> useful for other VOs.
>>>>
>>>> Atlas have some scripts - available under an open source licence that
>>>> allow a syncat dump to be checked against Atlas's LFC.
>>
>> https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsScripts
>>
>>>>
>>>> 1) They should be modified to work for any VO - and not rely on the
>>>> atlas specific stuff.
>>> Yes this sounds about right, the problem with the atlas scripts is that
>>> they rely on dq2. Since it is only an LFC frontend, it probably wouldn't
>>> be to arduous to reverse this dependency in order to make the scripts
>>> generic.
>>>
>>>>
>>>> 2) They should be extended so that rather than requiring all the manual
>>>> heavy lifting of sending syncat dumps around, they are able to talk to
>>>> SEs directly. Clearly this will be slower due to the SE overhead, but
>>>> the main problem with sending syncat dumps around is the significant
>>>> human resource it takes.
>>> Can't comment about this, I'm afraid I have no idea what making a syncat
>>> dump entails.
>>
>> Syncat is a standard format for dumps of what files are stored on an SE.
>> It's an XML format.  Making a syncat dump is relatively easy (for a
>> site), the challenge is getting 17 sites to respond in a timely manner,
>> run the right commands, send you the dump and so on.
>>
>> There's perhaps some argument that some python syncat classes would be
>> useful (and may even be provided by those atlas scripts).
>>
>> I've briefly discussed this with Sam Skipsey (ccd), who can probably
>> provide advice on the technical details of what would need to be done.
>>
>> Chris
>