Hi Alastair,
are you talking T2-T2 transfers? or T2-T3?
cheers
alessandra
Alastair Dewhurst wrote:
> Hi
>
> The following is an idea in the early stage of development, thoughts
> would be welcome.
>
> Currently if ATLAS users wish to get data that has been produced on
> the grid they use dq2-get (this basically just looks up where ATLAS
> thinks the file is and does a lcg-cp command). This is fine if the
> user has a small amount of data and a small number of files. However
> it is becoming apparent that users are producing quite large sets of
> output files (~1000 files and ~100GB). This is only likely to get
> worse as we get more real data. While dq2-get is fine for small
> amounts of data it becomes rather slow and unreliable on this scale.
> It can take users days to download their datasets and even longer to
> check to make sure that all the files were downloaded correctly (and
> not duplicated!).
>
> The suggestion is to tell users who want to get large amounts of user
> data to submit a datri (data transfer) request to copy the data to the
> tier 2 site they actually work at. If the site was to allow it they
> could then access the files directly from the storage element. For
> example at RALPP. I could request my dataset be moved to scratchdisk
> and then once it was there access it by looking in:
> /pnfs/pp.rl.ac.uk/data/atlas/atlasscratchdisk/
> I would use local dcache protocal to copy it out.
>
> The datri request has the advantage that it can be schedule, should be
> more efficient than a dq2-get command, will automatically retry
> failures. Of course for this to work the user would need to be able
> to access the local mass storage and it would be understandable if
> site admins didn't want this. However I am also aware that at some
> sites, local users already have limited access to the storage elements
> to give them somewhere to store their data offline.
>
> Comments?
>
> Alastair
>
--
The most effective way to do it, is to do it. (Amelia Earhart)
Northgrid Tier2 Technical Coordinator
http://www.hep.manchester.ac.uk/computing/tier2
|