So many streams to the RAID controller is bound to be a disaster. We
need to reduce the number
of parallel streams being generated by FTS/dCache.
Regards
Andrew
> -----Original Message-----
> From: GRIDPP2: Deployment and support of SRM and local
> storage management [mailto:[log in to unmask]] On
> Behalf Of Kostas Georgiou
> Sent: 20 July 2005 12:09
> To: [log in to unmask]
> Subject: Parallel Transfers
>
>
> On Tue, Jul 19, 2005 at 05:44:18PM +0100, Owen Synge wrote:
>
> > I think the disk thrashing issue is worth suggesting to others as a
> > performance hit as the experiments are trying to break
> things to find
> > out what breaks under what circumstances and then I think trying to
> > find out what is the best way to work with the software stack
> >
>
> Well i have parallel streams set to 1 for srm and it seemed
> to work fine for the Phedex transfers from RAL
> (~480/Mbit/sec). Somehow the SC3 transfers that Derek is
> running at the moment use something between 2 and 5 streams
> (from my strace logs) the end result is that we haven't
> managed to get more than ~80Mbit/sec :(
>
> From the strace logs it looks like each thread in d-cache
> writes it's own stream as it arrives instead of merging
> everything back in a buffer resulting in writes
> like:
>
> lseek(23, 106792960, SEEK_SET) = 106792960
> write(23, ..., 10240) = 10240
> ..
> lseek(23, 106844160, SEEK_SET) = 106844160
> write(23, ..., 10240) = 10240
>
> <guesswork>
> The OS/Raid controller might be able to merge everything back
> together before writing to the disk but with 250 streams that
> we have at the moment i think it's unlikely to happen (iostat
> reports minimal merges compared to writes).
>
> Since we are using RAID5 for the disks with a stripe of 64K
> the non merged 10K writes result in partial stripe write
> which causes Read-Modify-Write operations slowing down
> everything even more :( </guesswork>
>
> Too bad there is no source available to play with different
> settings :( I'll boot one of the pool nodes with the
> Anticipatory elevator which might be able to do better than
> the other ones but i don't expect it to do much difference :(
>
> Cheers,
> Kostas
>
|