Hi Tony,

Ah, the "you do it better" defence.

I administer a T2, it is not my job to deal with T1s and the T0, nor to write middleware, so if you need different bits of software for different Tiers that's CMS' problem and CMS a should deal with it. We already donated about 12 (wo)man hours to the project (because that's how long it took to find that missing perl rpm (which blocked the project at several sites for weeks on end, no?)), I'm afraid this is where it ends.

A storage dump is a debugging tool, if you need that on a weekly basis you are solving the wrong problem.

You might also want to rethink how you approach the sites, because a statement like this:
"a) is entirely up to you. We've explicitly avoided
prescribing how it should be done because we want sites to be able to do
whatever is most efficient for their storage."

is easily misunderstood as "we weren't able to write the code, but if we get 100 (or however many sites CMS has) sysadmins to do the work for us we'd be home free" [*]

I also find it hard to reconcile lofty statements about WLCG standards with instructions starting with:

"Get the latest version of PhEDEx code:

 git clone https://github.com/dmwm/PHEDEX.git

because clearly versioning is not part of any software releases according to WLCG standards

(maybe it's not, but ... arrghhh... why am I even going there ?).

[*] To quote the CMS spacemon webpage:
"There you will find instructions and scripts developed by CMS/ATLAS site admins and/or references for the tools provided with the storage solutions, as well as sample storage-dumps."

and
"Please do not hesitate to contribute your tools and bug-fixes to the common repository. You can fork the repository and make a pull request to merge your branch, or you can ask Eric, Giulio, Tony or Nicolo' for write-access."
You can see where I get that impression from.

Anyway, I am going to stop ranting now, I'm under no illusions about my impact here. You've heard from two site admins who aren't happy with how this is implemented, do with that information what you want, I'm going back to my day job ("keeping the lights blinking") now.

Daniela

On 30 June 2015 at 14:29, Tony Wildish <[log in to unmask]> wrote:

Hi Daniela,

can you name an already implemented interface that works with _all_ CMS
sites please? One that scales from the smaller T2's up to the T0?

And can you tell me how that would make things any easier? The SpaceMon
project is about monitoring space on SE's, which means it needs two
components: a) something to pull the metadata out of the SE about what's
on disk and b) tools to upload that to the database where we can analyse it.

How you do a) is entirely up to you. We've explicitly avoided
prescribing how it should be done because we want sites to be able to do
whatever is most efficient for their storage. All we care about is that
the information be available in a compact form for b), for which we
provide the tools. For that, we provide tools based on WLCG standards,
not CMS-specific.

Note that simply giving us a raw dump of the SE metadata won't satisfy
_your_ requirements. We don't care, but you, the site admins, don't want
to give us details about each users' files, or about files belonging to
ATLAS. That's where it gets messy.

Cheers,
Tony.

On 6/30/15 3:04 PM, Daniela Bauer wrote:
> Pick an already implemented interface to all these SEs and start there ?
> What is so wrong with that suggestion ?
>
>
> Daniela
>
>
> On 30 June 2015 at 13:51, Tony Wildish <[log in to unmask]
> <mailto:[log in to unmask]>> wrote:
>
> Hi,
>
> we're open to suggestions. So far, we haven't received any.
>
> Cheers,
> Tony.
>
> On 6/30/15 2:39 PM, L Kreczko wrote:
> > My experience is similar to Daniela's.
> > There must be better way's to do this.
> >
> > On 30 June 2015 at 13:33, Daniela Bauer
> > <[log in to unmask]
> <mailto:[log in to unmask]>
> > <mailto:[log in to unmask]
> <mailto:[log in to unmask]>>> wrote:
> >
> > Having produced one of these storage dumps, I'd like to add to the
> > record, that both the idea and implementation of this are truly
> > awful. There's nothing like a big experiment going around and making
> > each site implement something extra special just for them, because
> > they can't keep track of their data and/or use the existing
> > interfaces. It didn't help that the upload script didn't run on half
> > the sites to begin with until Simon debugged it (because he can't
> > help himself).
> >
> > Daniela
> >
> > On 30 June 2015 at 13:14, Brian Davies <[log in to unmask] <mailto:[log in to unmask]>
> > <mailto:[log in to unmask] <mailto:[log in to unmask]>>> wrote:
> >
> > Interesting cms link____
> >
> > __ __
> >
> > https://twiki.cern.ch/twiki/bin/view/CMSPublic/SpaceMonSiteAdmin____
> >
> > __ __
> >
> >
> >
> >
> > --
> > Sent from the pit of despair
> >
> > -----------------------------------------------------------
> > [log in to unmask] <mailto:[log in to unmask]>
> <mailto:[log in to unmask]
> <mailto:[log in to unmask]>>
> > HEP Group/Physics Dep
> > Imperial College
> > London, SW7 2BW
> > Tel: +44-(0)20-75947810 <tel:%2B44-%280%2920-75947810>
> <tel:%2B44-%280%2920-75947810>
> > http://www.hep.ph.ic.ac.uk/~dbauer/
> >
> >
> >
> >
> > --
> > *********************************************************
> > *Dr Lukasz Kreczko
> > Research Assistant*
> >
> > University of Bristol
> > HH Wills Physics Lab
> > University of Bristol
> > Tyndall Avenue
> > Bristol
> > BS8 1TL
> >
> > +44 (0)117 928 8724 <tel:%2B44%20%280%29117%20928%208724>
> > [log in to unmask] <mailto:[log in to unmask]>
> <mailto:[log in to unmask] <mailto:[log in to unmask]>>
> > *********************************************************
>
>
>
>
> --
> Sent from the pit of despair
>
> -----------------------------------------------------------
> [log in to unmask] <mailto:[log in to unmask]>
> HEP Group/Physics Dep
> Imperial College
> London, SW7 2BW
> Tel: +44-(0)20-75947810
> http://www.hep.ph.ic.ac.uk/~dbauer/

Sent from the pit of despair

-----------------------------------------------------------
[log in to unmask]
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: +44-(0)20-75947810
http://www.hep.ph.ic.ac.uk/~dbauer/