Monday, December 08, 2008

Solaris Link Aggregation

Setting up link aggregation in Solaris is pretty simple. First make sure you have a recent version (I'm using update 5 aka 5/08 and update 6 aka 10/08).

Then make sure your switch is configured. For example, on one of my Summit switches where I'm going to aggregate ports 7 and 8:

enable sharing 7 grouping 7 8 lacp

You can see the state of the network interfaces on the host using dladm, for example:

# dladm show-dev
nxge0 link: up speed: 1000 Mbps duplex: full
nxge1 link: up speed: 1000 Mbps duplex: full
nxge2 link: unknown speed: 0 Mbps duplex: unknown
nxge3 link: unknown speed: 0 Mbps duplex: unknown

Then on the host (connected to the console - trying to do this over the network is obviously going to be difficult), take down the existing interface:

ifconfig nxge0 down unplumb

Create an aggregate out of nxge0 and nxge1, with index 1 (why normal interfaces start out with index 0 and aggregations start out at 1 is one of those oddities):

dladm create-aggr -P L2 -l passive -d nxge0 -d nxge1 1

And then bring the interface up:

ifconfig aggr1 plumb
ifconfig aggr1 inet netmask broadcast up
and then rename /etc/hostname.nxge0 to /etc/hostname.aggr1 so the right thing happens next boot.

Here I've enabled LACP (the '-l passive' flag). I'm not absolutely sure how vital this is, but I think the switch and the host need to be set compatibly.

I had a little play with the policy. In the command above it's set to 'L2'. This didn't work well for me - all the traffic went down one of the links. Same with 'L3'. Setting to to use both L2 and L3 seemed to work better

dladm modify-aggr -P L2,L3 1
and I got traffic using both links, and an aggregate throughput obviously in excess of a single gigabit.

Monitoring the aggregate can again be done using dladm. For example, you can watch the traffic and how much goes down each link with 'dladm show-aggr -s -i 1'.


sgk said...

Hey, I was wondering how to do this just last nite. Thanks, you are a master computer psychic.

Anonymous said...

Very usefull thank you

Anonymous said...

Thank you! very usefull

Anonymous said...

Thank you! very usefull

Anonymous said...

Thank you! Very useful!

Ciccio said...

in Solaris 11.2, the -s flag for dladm show-aggr is no longer available.
Any suggestions on how to monitor the aggregate in this case?

Thanks a mill.

Peter Tribble said...

In Solaris 11.2, you probably want to investigate the dlstat command. (It's going to take a little playing with to get the information you want, and to be honest it might be easier to poke around at the underlying kstats.)