On 28/01/13 13:13, Wahid Bhimji wrote:
>   Hi
>
> DPM yaim doesnt set anything for those variables anymore. (Personally I
> think it should set something if there is nothing already set but thats
> just me).
>
> Indeed I have also had the settings for some time as I can imagine have
> most people (though still worth a reminder)

Indeed. It's my secret behind fast network rates.

I haven't changed it after upgrading to 10Gig. Perhaps I should. On se03 
  - the storage element that also acts as a gridftp machine, the setting is:

####################### from fasterdata.es.net
# increase TCP max buffer size setable using setsockopt()
# 16 MB with a few parallel streams is recommended for most 10G paths
# 32 MB might be needed for some very long end-to-end 10G or 40G paths
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
# (only change the 3rd value, and make it 16 MB or more)
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# recommended to increase this for 10G NICS
net.core.netdev_max_backlog = 30000

I also load the tcp_htcp module:

[root@se03 ~]# lsmod | grep tcp
tcp_htcp               37313  362

I'm currently using the ixgbe

[root@se03 ~]# lsmod | grep ixgbe
ixgbe                 160469  0
8021q                  58449  1 ixgbe
dca                    41605  1 ixgbe

For completeness, though it isn't clear it makes much difference,I'm 
using a later version of the ixgbe drivers.

[root@se03 ~]# modinfo  ixgbe | grep version
version:        3.10.17


> It was the "linux tuning expert" page that I noticed this time:
> http://fasterdata.es.net/host-tuning/linux/expert/
> (Not that that changes anything).
>
> Also the /sbin/ifconfig ethN txqueuelen 10000
> step, I don't think I ever did for some reason (and I'm trying to
> remember if it was a good one).
>
> on tcp_sack the advice is slightly ambiguous.


QMUL doesn't touch it FWIW.

Chris
>
> Wahid
>
> On 28 Jan 2013, at 12:33, Alessandra Forti <[log in to unmask]
> <mailto:[log in to unmask]>> wrote:
>
>> Hi,
>>
>> we discussed fasterdata.es.net <http://fasterdata.es.net> suggestions
>> long time ago when we had problems with XFS on SL5 and we didn't know
>> why yet.
>> I mentioned them in two talks one at the atlas software week [1] and
>> one at DPM community forum [2]. Chris has also been a supporter.
>> I haven't checked yet if DPM YAIM still overrides them though (should
>> do that now).
>>
>> cheers
>> alessandra
>>
>> [1]
>> http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=6&cad=rja&sqi=2&ved=0CE8QFjAF&url=http%3A%2F%2Findico.cern.ch%2FmaterialDisplay.py%3FcontribId%3D92%26sessionId%3D11%26materialId%3Dslides%26confId%3D119169&ei=FG0GUbjTKsTb0QXIj4HgCg&usg=AFQjCNHzj-lWF-ayfbQz7BDxQX6L-Zu2_A&sig2=BvbNuPUyZpEKAcWqKo_4zw
>> <http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=6&cad=rja&sqi=2&ved=0CE8QFjAF&url=http%3A%2F%2Findico.cern.ch%2FmaterialDisplay.py%3FcontribId%3D92%26sessionId%3D11%26materialId%3Dslides%26confId%3D119169&ei=FG0GUbjTKsTb0QXIj4HgCg&usg=AFQjCNHzj-lWF-ayfbQz7BDxQX6L-Zu2_A&sig2=BvbNuPUyZpEKAcWqKo_4zw>
>>
>> [2]
>> http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&sqi=2&ved=0CEkQFjAE&url=http%3A%2F%2Findico.cern.ch%2FmaterialDisplay.py%3FcontribId%3D4%26materialId%3Dslides%26confId%3D172988&ei=FG0GUbjTKsTb0QXIj4HgCg&usg=AFQjCNE-LS1XOXbakEJJicp-n9-ce_PaHg&sig2=i1IKpRHbspim_1BxzNVHug
>> <http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&sqi=2&ved=0CEkQFjAE&url=http%3A%2F%2Findico.cern.ch%2FmaterialDisplay.py%3FcontribId%3D4%26materialId%3Dslides%26confId%3D172988&ei=FG0GUbjTKsTb0QXIj4HgCg&usg=AFQjCNE-LS1XOXbakEJJicp-n9-ce_PaHg&sig2=i1IKpRHbspim_1BxzNVHug>
>>
>>
>> On 28/01/2013 12:15, Wahid Bhimji wrote:
>>>
>>> Hi
>>>
>>> I was just rechecking the TCP tuning on the ECDF pool servers and
>>> noticed the "advanced" guide on fasterdata.es.net
>>> <http://fasterdata.es.net/> which I don't remember looking at before.
>>> Anyway I thought I would reopen the question of what people are
>>> actually using.
>>> What I have is basically that from the non-advanced guide (pasted below):
>>> http://fasterdata.es.net/host-tuning/linux/ except
>>> a) We have those "optimised for network paths up to 100ms RTT" rather
>>> than 200 ms RTT (on 10 Gig NIC)
>>> b) I didn't ever do /sbin/ifconfig ethN txqueuelen 10000
>>>
>>> Do other people use these first set of values?
>>> There is also the question of tcp_sack which I have at the default (1)
>>>
>>> Cheers for any futher tips
>>>
>>> Wahid
>>>
>>> # increase TCP max buffer size setable using setsockopt()
>>> net.core.rmem_max = 16777216
>>> net.core.wmem_max = 16777216
>>> # increase Linux autotuning TCP buffer limit
>>> net.ipv4.tcp_rmem = 4096 87380 16777216
>>> net.ipv4.tcp_wmem = 4096 65536 16777216
>>> # increase the length of the processor input queue
>>> net.core.netdev_max_backlog = 30000
>>> # recommended default congestion control is htcp
>>> net.ipv4.tcp_congestion_control=htcp
>>>
>>>
>>>
>>> The University of Edinburgh is a charitable body, registered in
>>> Scotland, with registration number SC005336.
>>
>>
>> --
>> Facts aren't facts if they come from the wrong people. (Paul Krugman)
>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.