Hi,
As we host almost exclusively ATLAS files, I'm looking for the differences
between the ATLAS catalogue and the files on disk, to clean unregistered files
and delete catalogue entries that the corresponding file is missing.
David
On Wednesday 08 January 2014 10:15:42 Sam Skipsey wrote:
> Well, it's also complicated by the fact that "catalogue" can mean more
> than one thing (David, are you referring to the DPM catalogue, or an
> external catalogue, like ATLAS's DQ2/Rucio catalogue, or an LFC?).
>
> ATLAS's "dark data" tools care about the discrepancy between ATLAS
> Catalogues and the DPNS Catalogue.
>
> They do not (and cannot) identify discrepancies between the DPNS
> Catalogue and the actual files on disk.
>
> There *are* low level tools for checking DPNS v filesystem
> discrepancies (in the dpm-admin-tools package), but they deliberately
> do not actually perform modifications to the DPNS Catalogue (because
> it is unsafe to do this kind of thing without human intervention).
>
> So. Can I have some clarification as to what you'd like a tool to do?
>
> Sam
>
> On 7 January 2014 20:31, Christopher J. Walker <[log in to unmask]>
wrote:
> > On 07/01/14 18:25, David Cohen wrote:
> >> Hi,
> >> Does any one knows of a script that can read site parameters (e.g. space
> >> tokens, mount points SE endpoint) and run a dark data and catalogue
> >> clean-up,
> >> that I can run periodically and keep my storage clean and healthy ?
> >
> > For ATLAS, we (or more precisely colleagues at RAL) use ATLAS scripts -
> > which I think are here:
> >
> > https://twiki.cern.ch/twiki/bin/view/AtlasComputing/DDMOperationsScripts#D
> > etection_and_deletion_of_files
> >
> > You need data in syncat XML format. I've got a tool that will generate
> > this
> > format for StoRM. I'll send you a copy - it's a bit scrappy, but will do
> > the job (patches to clean it up welcome).
> >
> > For other VOs (and specifically non-LHC VOs), it would be really nice if
> > someone had such a tool - but nobody seems to have written one yet.
> >
> > Chris
> > --
> > Dr Christopher J. Walker
> > Queen Mary University of London
|