On 18/02/13 12:25, Alessandra Forti wrote:
> Hi Ewan,
> 
> we have those sysctl parameters as well. 

Can you (and Ewan) confirm that you have the same for:

sysctl net.ipv4.tcp_congestion_control
sysctl net.ipv4.tcp_sack
sysctl net.ipv4.tcp_dsack


BNL has lots of bandwidth and is a large distance away. Which other
distant site that has decent networking did you try? If you don't have
one, then AGLT2 or MWT2 might be appropriate - I know that Shawn McKee
<[log in to unmask]> takes a big interest in networking issues and
umich.edu is in one of those Tier-2s.



> I've played a bit with them and
> if I reduce them to the settings DPM used to set the 9stream transfers
> don't change and 1-2 streams get degraded and transfers take twice as
> long and if I instead double the values the results are in line as if I
> hadn't change them. Only one 9stream transfer went as fast as the
> 1-2stream and I don't think it was due to the sysctl parameters because
> the other 14 went as slow as usual.
> 
> This is a snapshot of a test I did this morning. I'm using all 5 files
> used by the sonar tests which are distributed on 5 different machines 2
> of these machines - they corresponding to the first two lines of each
> bunch - have double values (i.e. 33MB) settings.
> 
> repeating tests every few minutes doesn't help because I believe the
> files get eventually all in memory and 1-2 streams go eventually faster
> but not 9 stream ones.
> 
> 9 streams:
> 
>      47054848 bytes    511.15 KB/sec avg    522.27 KB/sec inst
>      30539776 bytes    498.73 KB/sec avg    513.71 KB/sec inst
>      23461888 bytes    383.14 KB/sec avg    386.98 KB/sec inst
>      30277632 bytes    495.28 KB/sec avg    498.26 KB/sec inst
>      29491200 bytes    480.80 KB/sec avg    507.73 KB/sec inst
> 
> 2 streams:
> 
>    1777729536 bytes  28934.40 KB/sec avg  31978.57 KB/sec inst
>    1776025600 bytes  28858.57 KB/sec avg  25437.87 KB/sec inst
>    1261230486 bytes  41055.68 KB/sec avg  41055.68 KB/sec inst
>    1354288154 bytes  44084.90 KB/sec avg  44084.90 KB/sec inst
>    2000000000 bytes  32071.02 KB/sec avg  23708.53 KB/sec inst
> 
> 1 stream:
> 
>     977272832 bytes  31812.27 KB/sec avg  31812.27 KB/sec inst
>     515768320 bytes  16789.33 KB/sec avg  16789.33 KB/sec inst
>     741832146 bytes  24148.18 KB/sec avg  24148.18 KB/sec inst
>     348258304 bytes  11336.53 KB/sec avg  11336.53 KB/sec inst
>     612237312 bytes  19996.25 KB/sec avg  19996.25 KB/sec inst
> 
> 
> Yes, we can ask to reduce the number of streams however some sites like
> QMUL do benefit from the 9 streams and sites like Lancaster, Glasgow and
> Liverpool although with some up and downs don't seem to suffer from this
> problem we should try to understand why it doesn't work at all sites for
> DPM.

That's great data. Can you let us know the command you used so we can
replicate it.

Have you tried with values between 2 and 9 streams (8 specifically). If
8 was somehow much better for you, and didn't make much difference for
QMUL and other sites, then it might tell us something.

Network analysis with wireshark does seem another way forward.

Chris
> 
> cheers
> alessandra
> 
> 
> 
> 
> 
> On 17/02/2013 16:09, Ewan MacMahon wrote:
>>> -----Original Message-----
>>> From: GRIDPP2: Deployment and support of SRM and local storage
>>> management
>>> [mailto:[log in to unmask]] On Behalf Of Alessandra Forti
>>>
>>> Since Manc degraded with the upgrade we should look at the OS/DPM
>>> configuration.
>> Oxford is now EMI2 DPM on SL5 across the whole system, but we've
>> seen the problem in the past when we still had gLite 3.2 pool nodes
>> with an EMI1 head node. I think the only time we weren't aware of
>> the problem (which could equally well be because it didn't exist,
>> or that I did and we just didn't know) was when we had SL4 gLite 3.1
>> pool nodes.
>>
>> We also have the 'fasterdata' TCP tweaks (I've copied the relevant bit
>> of our sysctl.conf below).
>>
>> If we think this is related to the 'extra' streams, can we tweak the
>> BNL FTS transfers to just use one stream? I know that ought to be
>> less efficient under normal circumstances, but it might be interesting
>> to see what it does here.
>>
>> Ewan
>>
>>
>>
>>
>> # Fromhttp://fasterdata.es.net/host-tuning/linux/
>> # increase TCP max buffer size setable using setsockopt()
>> # 16 MB with a few parallel streams is recommended for most 10G paths
>> # 32 MB might be needed for some very long end-to-end 10G or 40G paths
>>
>> # Commented values are OS defaults, uncommented ones are fasterdata ones,
>> # Defaults tested between 20121018 and 20121105 to no obvious effect -
>> Ewan
>>
>> net.core.rmem_max = 16777216
>> #net.core.rmem_max = 131071
>> net.core.wmem_max = 16777216
>> #net.core.wmem_max = 131071
>>
>> # increase Linux autotuning TCP buffer limits
>> # min, default, and max number of bytes to use
>> # (only change the 3rd value, and make it 16 MB or more)
>>
>> net.ipv4.tcp_rmem = 4096 87380 16777216
>> #net.ipv4.tcp_rmem = 4096 87380 4194304
>> net.ipv4.tcp_wmem = 4096 65536 16777216
>> #net.ipv4.tcp_wmem = 4096 16384 4194304
>>
>> # recommended to increase this for 10G NICS
>>
>> net.core.netdev_max_backlog = 30000
>> #net.core.netdev_max_backlog = 1000
> 
>