Hi Oxana,
You raise a number of issues here, about the perception of the
experiments use of the grid, but I'm afraid I can't agree with your
solution!
Oxana Smirnova wrote:
> Dear sysadmins,
>
> as several people pointed out, there is lack of understanding why do
> experiments use Grid. And since sysadmins are not members of all the
> experiment-specific mailing lists, they have little chance of learning
> it. I feel like I have to clarify several points:
>
> 1. Mailing lists are typically opened for posting only by the members,
> to protect it from SPAM. So please *join* [log in to unmask] or such
> before posting there.
>
> 2. Experiment representatives do not usually read LCG-ROLLOUT mailing
> list, because it is expected to deal with, well, LCG rollout.
Well, for sure. The problem is there has been no formal structure for
how site admins are to contact experiments. If Atlas wishes the formal
structure to be:
"Join [log in to unmask] and post message there."
then OK, that is Atlas's choice. (Although we were actually just told
"mail [log in to unmask]" :-) ) However, I do not personally feel this
is the correct solution. I'm not sure I need the information on this
list, and being asked to subscribe to yet-another-mailing-list per VO I
support doesn't seem right. All I need is a contact address I can mail
to when things go wrong...
(Also, given there are 80 odd sites in LCG, this means the
[log in to unmask] will now gain at least 80 new members, who aren't
necessarily members of atlas... if you want to turn [log in to unmask]
into a sys-admins discussion group - which is what we always seem to do
when we're given a new mailing list - then this would be the way!)
> 3. Experiments do run in a production mode. The recent LCG review stated
> that the LCG is far from being in a production state, but I'd like to
> see it proven wrong. So please don't be surprised to see files arranged
> "as if they run production". It's not "if", they *DO* run production,
> and each bit of data is needed in long term (months and years).
Absolutely, on this I completely agree with you! But I do wonder is it
really the case that all the data is actually still needed on the SE at
the end of the job? There were no input files that could be deleted?
No files that could be staged somewhere else?
> 4. Experiments perceive Storage Elements as *STORAGE* elements. They
> expect the files to stay there. And they can not move them elsewhere.
> Please do not advertise your facility as a Storage Element if it is
> *NOT* a Storage Element.
>
> 5. When your Storage Element is full, remove it from the Information
> System, or press the Data Management folks to develop a better solution.
> Do not stop PBS queues, as the jobs may well write to another Storage
> Element, not yours.
OK, this is where I cannot follow your reasoning.
OK, if I remove my SE from the Information System, then maybe you can't
get *any* of your data on my SE. Neither can any of the other VO's.
You can't even clean data off my SE that you don't need.
If that's an acceptable solution to you, then you clearly don't need
your data and I would be justified in doing rm -rf
/my/data/storage/atlas/*
But of course I am not serious and know this is not the case!
The point is that taking the SE out of the information system does not
solve the problem.
Pressing the data management folks to develop a better solution is
clearly the long term solution. But it hardly deals with the problem now?
> To summarise, we seem to be asked to move our files from nearly
> everywhere.
Yes, because Atlas have filled up a lot of SE's with your data. Clearly
Atlas have produced a lot of data on the grid, which is excellent, but
has not considered there to be any need to develop a coherent strategy
for managing that data, which is not excellent.
It is naive to claim "all SE's are equal" and assume that you can treat
a disk storage system at a minor regional centre as equally permanent
storage to a tape storage system at Tier 1 centres. It is simply
*false* to assume you can ignore the fact that there is no automatic
space management and just keep copying new data onto a finite disk
space. If that is Atlas' data mangement framework, then that framework
is clearly broken!
The current data management software is clearly missing a whole layer,
but if Atlas's response is to just close their eyes, refuse to manage
their use of disk resources and cause SE's to fill up and fall over,
that seems to me to be an abuse of resources.
To suggest a different perspective: given disk SE's will inevitably fill
up with data if the experiments do not implement their own
datamanagement frameworks, disk SE's should be considered *non-permanent
by default*. If you are unable to determine which SE's are disk or not,
then your data management framework will just have to assume any given
SE is non-permanent, and arrange for files to be transferred to known
SE's that you have discovered (by whatever means) to be permanent for you.
I guess everybody realizes it's very difficult, and BTW,
> we'll be producing 150% of what we have now in nearest 3 months. Thus a
> removal of 7% or what won't help. We really expected the storage
> facilities to be such, esp. Tier1-scale centers.
Then might I suggest you start taking a careful look at what actual
resources are available, find out where you have the capacity to
permanently store it, and develop a datamangement framework?
It does not seem to me to be beyond the capability of the current grid
tools to develop fairly simple processes for a job submission frameworks
to (a) transfer output data files to known permanent storage spaces
(such as tape storage at the Tier 1's) and (b) clean unneeded data files
off local SE's. It is certainly the case that other experiments have
managed this!
regards,
Owen.
--
=======================================================
Dr O J E Maroney # London Tier 2 Technical Co-ordinator
Tel. (+44)20 759 47802
Imperial College London
High Energy Physics Department
The Blackett Laboratory
Prince Consort Road, London, SW7 2BW
====================================
|
