Its not just min but also the step size of packet change. You are
lucky increasing the satrting value so high has not yet cuasewd you
any other issues. (I would be intersted to see what happens for local
WN jobs). I agree WAN tuning the WNs ( and the NAT box) is a good
idea. Instruction s for what works well can be found here:
http://fasterdata.es.net/TCP-tuning/
The tier1 saw problems with lo memeorry running out if you raise the
wuindow sizing too high.
for your 27ms link and 4Gb/s link. you would ideally have a window
size of ~ 14.5MB. Remember though that this should then be divided by
the number of concuurent data files that you have flowing. You may
probably already limited by what is sert at CERN. plus you end up
with issues when packets aren't alowed to shrink. Ie you may well
start to see issues if you were to have >296 transfers ( If i have
calculated it correctly.) What is th eping time between your WNs and
your storage nodes?
On 23 June 2010 12:54, Sam Skipsey <[log in to unmask]> wrote:
>
>
> On 23 June 2010 12:42, Christopher J.Walker <[log in to unmask]> wrote:
>>
>> Stuart Purdie wrote:
>>>
>>> To begin, the oblibitary pretty picture:
>>>
>>>
>>> http://lhcbweb.pic.es/DIRAC/LHCb-Production/visitor/systems/accountingPlots/dataOperation#ds9:_plotNames7:Qualitys9:_groupings6:Sources13:_timeSelectors2:-1s10:_startTimes10:2010-05-22s8:_endTimes10:2010-06-23s14:_OperationTypes14:putAndRegisters7:_Sources213:LCG.Barcelona.es,LCG.Bristol-HPC.uk,LCG.Bristol.uk,LCG.Brunel.uk,LCG.CNAF.it,LCG.Glasgow.uk,LCG.Liverpool.uk,LCG.PIC.es,LCG.Sheffield.uk,LCG.UKI-LT2-Brunel.uk,LCG.UKI-SCOTGRID-GLASGOW.uk,LCG.UNINA.it,LCG.UNIZAR.ess9:_typeNames13:DataOperatione
>>>
>>> (Hrm, that's a monster url: same thing at
>>> http://tinyurl.com/lhcbtransjune That is on fixed dates, not a rolling
>>> 'last month')
>>>
>>> What you're looking at is the transfer attempts + failures for LHCb
>>> traffic across a number of sites, for about the past month. Note that this
>>> is transfers, not jobs - a job can succeed after a couple of failed transfer
>>> attempts, so this is the most strict criterion to look at.
>>>
>>> I've included all the sites that I can see were having problems, along
>>> with PIC and CNAF to show the 'bad days' for comparison.
>>>
>>> The key thing to look is for Glasgow, after the 16th, when we switch from
>>> yellowish green (about 50%) to dark green (100% near enough). What changed
>>> was I tuned the TCP stack on the worker nodes. (Same thing YAIM does to DPM
>>> pool nodes). That resolved the problem.
>>>
>>> This the systcl parameters I set:
>>> # TCP buffer sizes
>>> net.ipv4.tcp_rmem = 131072 1048576 2097152
>>> net.ipv4.tcp_wmem = 131072 1048576 2097152
>>> net.ipv4.tcp_mem = 131072 1048576 2097152
>>>
>>> net.core.rmem_default = 1048576
>>> net.core.wmem_default = 1048576
>>> net.core.rmem_max = 2097152
>>> net.core.wmem_max = 2097152
>>>
>>> # SACK and timestamps - turn off
>>> net.ipv4.tcp_dsack = 0
>>> net.ipv4.tcp_sack = 0
>>> net.ipv4.tcp_timestamps = 0
>>>
>>
>> Can you follow up to the list with the previous values. It isn't clear
>> from your mail what you increased/decreased.
>>
>
> The defaults are:
> net.ipv4.tcp_rmem = 4096 87380 4194304
> net.ipv4.tcp_wmem = 4096 16384 4194304
> net.ipv4.tcp_mem = 196608 262144 393216
> net.core.rmem_default = 129024
> net.core.wmem_default = 129024
> net.core.rmem_max = 131071
> net.core.wmem_max = 131071
> net.ipv4.tcp_dsack = 1
> net.ipv4.tcp_sack = 1
> net.ipv4.tcp_timestamps = 1
> (basically, the min, starting and max values for tcp window size were all
> increased by ~x10 - to the YAIM tuned values for disk servers (which happen
> to be a good approximation to tuned for transfers to RAL and CERN) - and
> sack was turned off).
>
> Sam
>>
>> Chris
>>
>>>
>>> Ok, so that's the what - the why is not so clear. I was working on the
>>> theory that the presence of the NAT boxes represented a network
>>> inefficiency, and that if the transfers were given longer then they would
>>> compete successfully. Therefore the approach was to try to optimise the
>>> transfers from the worker nodes to CERN, so that if they went a bit quicker,
>>> they'd complete before the timeouts. Note that (at least for us), RAL is 12
>>> ms away, and CERN is 27 ms away. The closer one is to CERN, the smaller
>>> effect this change should have (we might well be in the worst case here, at
>>> least until UKI-SCOTGRID-SHETLAND gets off the ground).
>>> By tuning the worker node for a Long Fat Network, which that sort of
>>> connect is, we get more data moved faster. (Although the target nodes are
>>> tuned, TCP/IP is limited by the congestion window on both sides, hence
>>> tweaking the worker nodes as well.) I've been poking at other parameters as
>>> well, but the parameters above worked so well that I can't find any
>>> differences with any others. (It's also worth noting that these made no
>>> difference in transfers to or from our local SE - i.e. they don't seem to
>>> cause any problems even if not useful.)
>>>
>>> I'd be interested if applying this sort of tuning to worker nodes will
>>> have any effect at the other sites that are having transfer problems -
>>> Brunel, Liverpool, Sheffield and Bristol. Also, I'd be interested on the
>>> round trip times between the worker nodes and CERN (i.e. through the NAT) -
>>> I've been traceroute www.cern.ch, and reading off the last one I can.
>>> Raja - I note that Barcelona and UNIZAR both show similar (although less
>>> severe) effects as the UK. Your opposite number in Spain might be
>>> interested in this - certainly I'm curious about their configuration: I
>>> rather suspect they have NAT's and untuned worker nodes.
>>>
>>>
>
>
|