Hi,
If you're using layer2 hashing with LACP then a single host mac address
will always go through the same bonded interface. If your tests are
between just two machines you will see this. If your machines are
communicating over layer3 then this will always be the case unless you
use layer2+3 or layer3+4 hashing (but as you're using a Class B network
I expect you don't use any routing between hosts).
Try some more transfers from different hosts. The load should show up as
being spread across the different interfaces eg a bond on one of our
storage servers has:
eth1 Link encap:Ethernet HWaddr 00:15:17:78:78:51
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:97249688 errors:0 dropped:0 overruns:0 frame:0
TX packets:156006915 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:165088109873 (153.7 GiB) TX bytes:1232485987864
(1.1 TiB)
eth2 Link encap:Ethernet HWaddr 00:15:17:78:78:51
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:87233076 errors:0 dropped:0 overruns:0 frame:0
TX packets:159241749 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:171713805781 (159.9 GiB) TX bytes:1268111167924
(1.1 TiB)
eth3 Link encap:Ethernet HWaddr 00:15:17:78:78:51
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:94892104 errors:0 dropped:0 overruns:0 frame:0
TX packets:160132801 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:173511432378 (161.5 GiB) TX bytes:1266240156168
(1.1 TiB)
John
Andrew Elwell wrote:
> Hi Folks,
>
> Middleware related in that we're trying to use link aggregation on our
> new disk servers with SLC4X
>
> I've set up the servers as follows:
>
> /etc/modprobe.conf
> alias eth0 igb
> alias eth1 igb
> alias bond0 bonding
> options bond0 mode=4 miimon=100
>
> and /etc/sysconfig/network-scripts/ are:
> ifcfg-eth0
> DEVICE=eth0
> BOOTPROTO=none
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> ISALIAS=no
>
> ifcfg-eth1
> DEVICE=eth1
> BOOTPROTO=none
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> ISALIAS=no
>
> ifcfg-bond0
> DEVICE=bond0
> BOOTPROTO=none
> ONBOOT=yes
> IPADDR=10.141.245.48
> NETMASK=255.255.0.0
> USERCTL=no
>
>
> which all seem to come up OK
>
> disk048:~# ifconfig
> bond0 Link encap:Ethernet HWaddr 00:30:48:C3:26:7A
> inet addr:10.141.245.48 Bcast:10.141.255.255 Mask:255.255.0.0
> inet6 addr: fe80::230:48ff:fec3:267a/64 Scope:Link
> UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
> RX packets:242700760 errors:0 dropped:0 overruns:0 frame:0
> TX packets:103425281 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:367020039578 (341.8 GiB) TX bytes:5588397361 (5.2 GiB)
>
> eth0 Link encap:Ethernet HWaddr 00:30:48:C3:26:7A
> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
> RX packets:242700443 errors:0 dropped:0 overruns:0 frame:0
> TX packets:103423303 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:367020031506 (341.8 GiB) TX bytes:5588285549 (5.2 GiB)
> Memory:d8220000-d8240000
>
> eth1 Link encap:Ethernet HWaddr 00:30:48:C3:26:7A
> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
> RX packets:335 errors:0 dropped:0 overruns:0 frame:0
> TX packets:1987 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:35324 (34.4 KiB) TX bytes:112506 (109.8 KiB)
> Memory:d8260000-d8280000
>
>
> disk048:~# cat /proc/net/bonding/bond0
> Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)
>
> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
> MII Status: up
> MII Polling Interval (ms): 100
> Up Delay (ms): 0
> Down Delay (ms): 0
>
> 802.3ad info
> LACP rate: slow
> Active Aggregator Info:
> Aggregator ID: 1
> Number of ports: 2
> Actor Key: 17
> Partner Key: 16432
> Partner Mac Address: 00:22:67:7d:c0:00
>
> Slave Interface: eth0
> MII Status: up
> Link Failure Count: 0
> Permanent HW addr: 00:30:48:c3:26:7a
> Aggregator ID: 1
>
> Slave Interface: eth1
> MII Status: up
> Link Failure Count: 0
> Permanent HW addr: 00:30:48:c3:26:7b
> Aggregator ID: 1
>
>
>
> from the switch end it looks reasonable too:
>
> stack02#show lacp port 3/27-28
> Admin Oper Trunk Partner
> Unit/Port Priority Lacp A/I Timeout Key Key AggrId Id Port
> Status
> --------- -------- ------- --- ------- ----- ----- ------ ----- -------
> ------
> 3/27 32768 Passive A Long 48 16432 8212 14 1
> Active
> 3/28 32768 Passive A Long 48 16432 8212 14 2
> Active
> stack02#show lacp aggr 8212
> Aggr ID: 8212
> Trunk: 14
> Status : Enabled
> Type : LA
> Actor Lag ID : 8000-0022677DC000-4030
> Partner Lag ID : FFFF-003048C3267A-0011
> Members : 3/27-28
> Standby Members: NONE
>
>
> stack02#show lacp stats 3/27-28
> Unit/Port 3/27 -------------------------------------
> LACPDUs Rx: 216
> LACPDUs Tx: 217
> MarkerPDUs Rx: 0
> MarkerResponsePDUs Rx: 0
> MarkerPDUs Tx: 0
> MarkerResponsePDUs Tx: 0
> UnknownPDUs Rx: 0
> IllegalPDUs Rx: 0
> Unit/Port 3/28 -------------------------------------
> LACPDUs Rx: 208
> LACPDUs Tx: 207
> MarkerPDUs Rx: 0
> MarkerResponsePDUs Rx: 0
> MarkerPDUs Tx: 0
> MarkerResponsePDUs Tx: 0
> UnknownPDUs Rx: 0
> IllegalPDUs Rx: 0
>
>
>
> but, despite all of the above the ifconfig shows that nearly all of the
> traffic is going through eth0 .. is this to be expected ot is it a
> side-effect of globus-gridftp-server?
>
> Hopefully there's some networky person on the list who can point me to
> any mistakes
>
> Many thanks
>
> Andrew
>
>
>
--
Dr John Bland, Systems Administrator
Room 220, Oliver Lodge
Particle Physics Group, University of Liverpool
Mail: [log in to unmask]
Tel : 0151 794 2911
"I canna change the laws of physics, Captain!"
|
