On Tue, Nov 16, 2010 at 00:10, John Gordon <[log in to unmask]> wrote:
> Can we not lobby ATLAS again to accept that the traffic to a site should not just be based on the amount of disk they have but some function of the network bandwidth and disk. One obviously can't just change from disk to bandwidth or a site with a fat pipe and not much disk would suffer a different fate.
Hi John
It would be really stupid to put disk and CPU at a site which was
connected down a soggy piece of string. QMUL will be ATLAS's second
largest T2 site so it needs better networking. I believe this is being
progressed.
At the moment Chris's solution seems technically the best one. FTS
just has no hooks to set a bandwidth cap on a channel and if we pared
the number of slots to the bone then we'd be killed by small file
overheads. Having gridftp set the TOS flag is sensible.
Please note that it is not because of PD2P that QMUL is getting a lot
of data. We are in the midst of a reprocessing campaign and most of
the data is moved by post-repocessing subscriptions (darker brown vs.
light brown in the plot).
Cheers
Graeme
>
>> -----Original Message-----
>> From: Testbed Support for GridPP member institutes [mailto:TB-
>> [log in to unmask]] On Behalf Of Christopher J.Walker
>> Sent: 15 November 2010 19:37
>> To: [log in to unmask]
>> Subject: GridFTP ToS (and Traffic shaping/policing)
>>
>> After the spring reprocessing, ATLAS maxed out QMUL's link to Janet for
>> a couple of weeks. This seemed to cause job unreliability - presumably
>> they were unable to phone home in the face of large amounts of packet loss.
>>
>> With the recent reprocessing and/or Atlas's move to pd2p, they are again
>> transferring lots of data to QMUL, and again maxing out our link.
>>
>> I've ended up implementing traffic policing on our SE. What we do is
>> drop traffic to gridFTP's data transfer port range when we fill 75% of
>> the link. This causes TCP/IP to backoff - so leaving some space on the
>> link.
>>
>> We are currently only dropping incoming packets in the globus portrange.
>> This presumably includes ack packets for outgoing traffic which must
>> be a bad thing. We'll also need to drop outgoing traffic too I think[1].
>>
>> One of the suggestions in the Linux Advanced Routing & Traffic Control
>> howto (http://lartc.org/) is to drop based on the TOS IP headers. This
>> doesn't seem to be set in gridftp packets - (though it does in scp
>> packets).
>>
>> TOS would seem an obvious thing to set - it would enable us and/or Janet
>> to drop the bulk data in preference to the interactive data. It would
>> also mean I can tell our network team to drop bulk packets rather than a
>> complicated host/port range. Does anyone know why globus don't set it?
>>
>> Chris
>>
>>
>> [1] Indeed globus provide a script to do this at
>> http://www.globus.org/toolkit/data/gridftp/bwlimit.html
>
|