Things are much better now that files can be staged one at a time. See
for example RHUL results in this test
http://gangarobot.cern.ch/hc/1017/test/
the CPU efficiency was about 91% and the event rate 18 Hz.
Duncan
Alessandra Forti wrote:
>> I was under the impression the advantages of merged files for other
>> tasks (eg FTS) outweighed the benefits of higher analysis efficiency and
>> we don't want them tiny because that can overload the head node with
>> multiple connections.
>
> transfers, head nodes load, tape sytems, bookkeeping, data
> management.... it all gets more complicated with many small files. Even
> analysis jobs can become a problem if the user doesn't analyse enough
> data per job, i.e. too many short jobs. Merging is not an atlas novelty,
> babar was already merging 6 years ago.
>
> cheers
> alessandra
>
> John Bland wrote:
>> Christopher J.Walker wrote:
>>
>>> I have long held that with the inefficient access pattern, life would be
>>> much better if the files were somewhat smaller (say 0.5Gig). Graeme told
>>> me last week I was the first person to ask for that, so I haven't been
>>> shouting loud enough.
>>>
>>
>> One would have thought that having smaller files would have been
>> self-evident, we already have HC results from before the merged files
>> became so big, the problems of the file access and how to overcome them
>> are well known and the problem of the large files hampering file caching
>> has been pointed out by me on at least a few occasions (perhaps not
>> explicitly).
>>
>>
>> If we can have smaller files, then please, make them smaller, because
>> this would have massive impacts on job efficiencies if we can get all
>> the files in RAM for a change. These initial results point to local file
>> access still being the best for ordered files but even with the optimal
>> file layout the absolute best place to have a local file is wholly in
>> RAM, not disk.
>>
>> If we're expected to have 2GB/core then the files should be < 2GB-(RSS
>> of typical analysis job). I guess this would be of the order of
>> 0.5-1.0GB ie a factor of 2-3x smaller than the typical AOD we've seen
>> which shouldn't unduly stress storage servers with extra connections but
>> would have a large effect on job efficiencies in a scalable manner.
>>
>> If files were wholly cached the ordering or otherwise of the file would
>> be of little consequence (in this regime), other than its reduction in
>> file size, and could have been done at any point in the past few years
>> with minimal impact.
>> John
>>
>>
>
|