Hi -
This use case actually came up at Birmingham last week. My first
suggested solution was that our users transfer their data (via a datri
request) to Birmingham and then access it via our own WNs. They weren't
overly enamoured with this solution because it would have meant waiting
for the transfer to finish. Plus, their output dataset is over 600GB in
size, which would have eaten a healthy chunk of our SCRATCHDISK.
In this particular case, they're iterating over an ntuple and outputing a
few simple numbers, so they decided to use ganga to submit ROOT jobs to
the different dataset locations, with the intention of dq2-get'ing the
smaller output (<100MB) after the jobs complete.
Chris
On Mon, 5 Jul 2010, Alastair Dewhurst wrote:
> Hi
>
> The following is an idea in the early stage of development, thoughts would be
> welcome.
>
> Currently if ATLAS users wish to get data that has been produced on the grid
> they use dq2-get (this basically just looks up where ATLAS thinks the file is
> and does a lcg-cp command). This is fine if the user has a small amount of
> data and a small number of files. However it is becoming apparent that users
> are producing quite large sets of output files (~1000 files and ~100GB).
> This is only likely to get worse as we get more real data. While dq2-get is
> fine for small amounts of data it becomes rather slow and unreliable on this
> scale. It can take users days to download their datasets and even longer to
> check to make sure that all the files were downloaded correctly (and not
> duplicated!).
>
> The suggestion is to tell users who want to get large amounts of user data to
> submit a datri (data transfer) request to copy the data to the tier 2 site
> they actually work at. If the site was to allow it they could then access
> the files directly from the storage element. For example at RALPP. I could
> request my dataset be moved to scratchdisk and then once it was there access
> it by looking in:
> /pnfs/pp.rl.ac.uk/data/atlas/atlasscratchdisk/
> I would use local dcache protocal to copy it out.
>
> The datri request has the advantage that it can be schedule, should be more
> efficient than a dq2-get command, will automatically retry failures. Of
> course for this to work the user would need to be able to access the local
> mass storage and it would be understandable if site admins didn't want this.
> However I am also aware that at some sites, local users already have limited
> access to the storage elements to give them somewhere to store their data
> offline.
>
> Comments?
>
> Alastair
--
West 326
Physics and Astronomy
University of Birmingham
Edgbaston
Birmingham
B15 2TT
(Office) 0121 414 4700
(Mobile) 0798 666 1959
|