Jensen, J (Jens) wrote:
> Now with minutes, too.
> http://indico.cern.ch/conferenceDisplay.py?confId=66531
> 
> Thanks once again for a useful meeting.  And special thanks to Brian for
> also keeping an eye on the slot, I was a bit later than usual having
> been on leave and only coming back late-ish last night.

Sorry I wasn't at the meeting, missed the email. I noticed in the 
minutes the subject of moving from rfcp to RFIO with 4kB buffers. We've 
been looking into the problems of rfcp at Liverpool and have come to 
some conclusions about the lack of performance and ways of improving 
efficiency. Unfortunately the improvements we've applied may only be 
applicable for a small number of sites. We've kept quiet about them 
while we got some real numbers from HC tests and until the site 
throughput reports were made public.

First an overview of the problem, as some of the inherent problems may 
not be known to everyone. [Wall of text begins]

Atlas analysis jobs need to read in events from data files. These data 
files now tend to come in small numbers of large files (multi-gigabyte).

The access patterns appear to be pseudo-random, in that the analysis 
processes seek back and forth across the datafile in big jumps in a (to 
the system) unpredictable manner. It is this pseudo-random nature of 
read access that is the root cause of all of the problems we've been 
trying to solve with buffers etc over the past year. This is made worse 
by the large size of the files.

Until this is changed in some significant way I don't see atlas analysis 
efficiencies ever being good without significant (over)investment in 
extra hardware. Local atlas people think this is down to the arrangement 
of the different types of atlas data within the ROOT file, rather than 
the way ROOT itself operates, but I'm certainly no expert on ROOT.

So far, there have been two main threads of access; direct POSIX IO to 
the storage servers and file staging data to the WN local disk. Each has 
inherent drawbacks. POSIX IO hits the extra latency of the access 
protocol, TCP/IP, ethernet and can saturate network bandwidth, and 
ultimately still ends up hitting a physical 'disk'. Local WN disk is 
relatively slow, possibly doing other things as well and is being shared 
between up to 8 processes on modern machines.

In either case whenever the event data is buffered/cached in system RAM 
efficiency is very high. As soon as access hits bare metal (disk or 
network) efficiency drops significantly as the access times are many 
orders of magnitude slower (eg RAM ns, disk/net ms).

With the current large files it is no longer possible to fully buffer 
the entire file in RAM with RFIO, meaning we're hitting the network for 
more reads rather than RAM. Large buffers pull in more data (in the hope 
of buffering the next events) than necessary and so lead to bandwidth 
saturation. Small (4kB) buffers only read what they need but they do hit 
the network latency problem for each and every read.

Those reads also translate to multiple random reads for the storage 
pools (as they cannot possibly cache all the files they are serving). 
Current RAID arrays only have a relatively small number of IO operations 
per second and so they will likely only scale so far before reaching the 
hardware limit of the array. The result is that job efficiencies are 
relatively stable but still quite low, and the overall site throughput 
may hit a brick wall when job numbers increase and the pool nodes 
saturate (depends on number of jobs and number of pools).

There may be a particular small buffer size that gives the best 
efficiency but I don't see any sort of magic bullet that massively 
increases the current maximum.

The other option of file staging also fares badly because of the access 
patterns. For one job the files are cached in RAM and efficiency is very 
high. Unfortunately there isn't generally enough free RAM on current 
8core systems to cache all of the files for 8 jobs (which translates to 
a minimum of 16 files, although currently jobs seem to be holding many 
more than this for some reason that hasn't been explained yet).

Sites have been seeing efficiencies dropping off for more than 4 jobs 
per node. Over this the system merely gets more and more bogged down 
with IO and efficiencies get worse.

Following system activity during analysis shows that while there is a 
lot of read access to the WN disk the major disk activity (at least 
twice as much and often more) is *write* access. See this plot for a WN disk

http://hep.ph.liv.ac.uk/~jbland/device_i_o_transfers__dev9_0.png

The 10s of GBs of files being read in have to be written to disk. A 
certain proportion of the files are still cached in RAM so those reads 
hit RAM not disk. The disk writes consume system resources and block reads.

There are also writes continually being made to disk for system logs, 
job stdout/stderr etc that also compete with job read access.

With this in mind we concentrated on increasing the WN disk write 
performance. There are three ways we did this on our systems

1) Convert our existing software RAID1 arrays to RAID0
2) Added a separate small system disk (for logs, job output etc)
3) Short stroked the 1TB drives using 64GB partitions

All of the above have shown improvements with local and HC tests as they 
all improve disk throughput and latency.

Another improvement can come from altering the behaviour of the page 
cache. By default the file data will be committed to physical disk if 
more than a certain percentage of the RAM is being used for cache, or if 
the file data is more than 30s old. This means any cached file will 
eventually be fully written to disk. While this may be useful for 
permanent files the atlas staged files are only transitory and don't 
need to be stored on disk, any write of staged files to disk is a waste 
of resources.

The page cache flushing can be altered through sysctl. We have changed 
this to a 30minute timeout (rather than 30s) and only write files when 
all RAM is used. This means only the parts of files that have to be 
written to disk (to make way for other processes or new files) are 
written, not all files all of the time. With local tests this made an 
improvement to throughput, but the current HC tests are keeping multiple 
files stored on the WN so the effect in this case is minimal (as all 
files have to be written eventually as there just isn't enough room).

The overall effect of these four methods in combination can be seen in 
this plot

http://hep.ph.liv.ac.uk/~jbland/tuned-atlas-throughput.png

Job efficiencies are higher than our standard setup and consequently 
throughput is higher. Throughput remains good up to 7 jobs/node now, and 
even 8 jobs/node is better than the best untuned throughput. I think 
that this would be even better if the HC file staging only kept two 
files/job on the WN if possible.

File staging should scale higher than RFIO, as the efficiency drop off 
is per node, the only limit after that is how many nodes can run before 
saturating the LAN or pool node bandwidth. It's unlikely to scale with 
cores/node, however.

 From comments made by other sites it might be that Liverpool are unique 
in using multiple disks already on the WNs, so the above has cost us 
very little (apart from the little system disks) and garnered big 
improvements. Other sites may not be able to make use of this tuning if 
they only have single disks or limited disk slots, although the page 
cache changes can be applied to any system on the fly.

As there is a big mix of hardware out there it is hard to provide a 
single solution that is best, sites with 8core systems may need a 
different access method to those with 2 or 4, and lustre adds an extra 
option that seems to do quite well. In a few years when SSDs are big and 
affordable enough the problem may go away (there are a lot of events to 
be processed between then and now, though).

I would look forward to an RFIO test to compare our current tuned file 
stager setup with small buffer RFIO (the way we're doing our tests make 
it hard to really compare with other site results).

John

-- 
Dr John Bland, Systems Administrator
Room 220, Oliver Lodge
Particle Physics Group, University of Liverpool
Mail: [log in to unmask]
Tel : 0151 794 2911
"I canna change the laws of physics, Captain!"