Hi Andrew,
Andrew Elwell wrote:
> Hi Folks,
>
> Middleware related in that we're trying to use link aggregation on our
> new disk servers with SLC4X
>
> I've set up the servers as follows:
>
> /etc/modprobe.conf
> alias eth0 igb
> alias eth1 igb
> alias bond0 bonding
> options bond0 mode=4 miimon=100
>
> and /etc/sysconfig/network-scripts/ are:
> ifcfg-eth0
> DEVICE=eth0
> BOOTPROTO=none
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> ISALIAS=no
>
> ifcfg-eth1
> DEVICE=eth1
> BOOTPROTO=none
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> ISALIAS=no
>
> ifcfg-bond0
> DEVICE=bond0
> BOOTPROTO=none
> ONBOOT=yes
> IPADDR=10.141.245.48
> NETMASK=255.255.0.0
> USERCTL=no
>
>
> which all seem to come up OK
>
> disk048:~# ifconfig
> bond0 Link encap:Ethernet HWaddr 00:30:48:C3:26:7A
> inet addr:10.141.245.48 Bcast:10.141.255.255 Mask:255.255.0.0
> inet6 addr: fe80::230:48ff:fec3:267a/64 Scope:Link
> UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
> RX packets:242700760 errors:0 dropped:0 overruns:0 frame:0
> TX packets:103425281 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:367020039578 (341.8 GiB) TX bytes:5588397361 (5.2
> GiB)
>
> eth0 Link encap:Ethernet HWaddr 00:30:48:C3:26:7A
> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
> RX packets:242700443 errors:0 dropped:0 overruns:0 frame:0
> TX packets:103423303 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:367020031506 (341.8 GiB) TX bytes:5588285549 (5.2
> GiB)
> Memory:d8220000-d8240000
>
> eth1 Link encap:Ethernet HWaddr 00:30:48:C3:26:7A
> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
> RX packets:335 errors:0 dropped:0 overruns:0 frame:0
> TX packets:1987 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:35324 (34.4 KiB) TX bytes:112506 (109.8 KiB)
> Memory:d8260000-d8280000
>
>
> disk048:~# cat /proc/net/bonding/bond0
> Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)
>
> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
> MII Status: up
> MII Polling Interval (ms): 100
> Up Delay (ms): 0
> Down Delay (ms): 0
>
> 802.3ad info
> LACP rate: slow
> Active Aggregator Info:
> Aggregator ID: 1
> Number of ports: 2
> Actor Key: 17
> Partner Key: 16432
> Partner Mac Address: 00:22:67:7d:c0:00
>
> Slave Interface: eth0
> MII Status: up
> Link Failure Count: 0
> Permanent HW addr: 00:30:48:c3:26:7a
> Aggregator ID: 1
>
> Slave Interface: eth1
> MII Status: up
> Link Failure Count: 0
> Permanent HW addr: 00:30:48:c3:26:7b
> Aggregator ID: 1
>
>
>
> from the switch end it looks reasonable too:
>
> stack02#show lacp port 3/27-28
> Admin Oper Trunk Partner
> Unit/Port Priority Lacp A/I Timeout Key Key AggrId Id
> Port Status
> --------- -------- ------- --- ------- ----- ----- ------ -----
> ------- ------
> 3/27 32768 Passive A Long 48 16432 8212 14
> 1 Active
> 3/28 32768 Passive A Long 48 16432 8212 14
> 2 Active
> stack02#show lacp aggr 8212
> Aggr ID: 8212
> Trunk: 14
> Status : Enabled
> Type : LA
> Actor Lag ID : 8000-0022677DC000-4030
> Partner Lag ID : FFFF-003048C3267A-0011
> Members : 3/27-28
> Standby Members: NONE
>
>
> stack02#show lacp stats 3/27-28
> Unit/Port 3/27 -------------------------------------
> LACPDUs Rx: 216
> LACPDUs Tx: 217
> MarkerPDUs Rx: 0
> MarkerResponsePDUs Rx: 0
> MarkerPDUs Tx: 0
> MarkerResponsePDUs Tx: 0
> UnknownPDUs Rx: 0
> IllegalPDUs Rx: 0
> Unit/Port 3/28 -------------------------------------
> LACPDUs Rx: 208
> LACPDUs Tx: 207
> MarkerPDUs Rx: 0
> MarkerResponsePDUs Rx: 0
> MarkerPDUs Tx: 0
> MarkerResponsePDUs Tx: 0
> UnknownPDUs Rx: 0
> IllegalPDUs Rx: 0
>
>
>
> but, despite all of the above the ifconfig shows that nearly all of
> the traffic is going through eth0 .. is this to be expected ot is it a
> side-effect of globus-gridftp-server?
read
http://garr.dl.sourceforge.net/sourceforge/bonding/bonding.txt
for details on channel bonding; in mode 4 (802.3ad) the default transmit
hash policy (xmit_hash_policy) is described as
xmit_hash_policy
Selects the transmit hash policy to use for slave selection in
balance-xor and 802.3ad modes. Possible values are:
layer2
Uses XOR of hardware MAC addresses to generate the
hash. The formula is
(source MAC XOR destination MAC) modulo slave count
This algorithm will place all traffic to a particular
network peer on the same slave.
This algorithm is 802.3ad compliant.
in order words, it is to be expected that all traffic to a single
gridftp server is going through eth0 ...
if the Nortel switch allows it you could try
'xmit_hash_policy=layter3+4' - this should divide the traffic over all
interfaces involved but this is not fully 802.3ad compliant.
cheers,
JJK / Jan Just Keijser
Nikhef Amsterdam
|