Hi
The following is an idea in the early stage of development, thoughts
would be welcome.
Currently if ATLAS users wish to get data that has been produced on
the grid they use dq2-get (this basically just looks up where ATLAS
thinks the file is and does a lcg-cp command). This is fine if the
user has a small amount of data and a small number of files. However
it is becoming apparent that users are producing quite large sets of
output files (~1000 files and ~100GB). This is only likely to get
worse as we get more real data. While dq2-get is fine for small
amounts of data it becomes rather slow and unreliable on this scale.
It can take users days to download their datasets and even longer to
check to make sure that all the files were downloaded correctly (and
not duplicated!).
The suggestion is to tell users who want to get large amounts of user
data to submit a datri (data transfer) request to copy the data to
the tier 2 site they actually work at. If the site was to allow it
they could then access the files directly from the storage element.
For example at RALPP. I could request my dataset be moved to
scratchdisk and then once it was there access it by looking in:
/pnfs/pp.rl.ac.uk/data/atlas/atlasscratchdisk/
I would use local dcache protocal to copy it out.
The datri request has the advantage that it can be schedule, should
be more efficient than a dq2-get command, will automatically retry
failures. Of course for this to work the user would need to be able
to access the local mass storage and it would be understandable if
site admins didn't want this. However I am also aware that at some
sites, local users already have limited access to the storage
elements to give them somewhere to store their data offline.
Comments?
Alastair
|