kern/185967: Link Aggregation LAGG: LACP not working in 10.0

[Ben]

Hi,

I upgraded from FreeBSD 9.2-RELEASE to 10.0-RELEASE. FreeBSD 9.2 was
configured to use LACP with two igb devices.

Now it stopped working after the upgrade.

This is a screenshot of ifconfig -a after the upgrade to FreeBSD
10.0-RELEASE: http://tinypic.com/view.php?pic=28jvgpw&s=5#.Uu9PXT1dVPM

A PR is currently open:
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/185967

It is set to low, but I would like somebody to have a look into it as it
obviously has a great influence on our infrastructure. The only way to
"solve" it is currently switching back to FreeBSD 9.2.

The suggested fix "use failover" seems not to work.

Thank you for your help.

Best regards
Ben
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net...@freebsd.org"

[Scott Long]

Hi,

You’re probably running into the consequences of r253687. Check to see the value of ‘sysctl net.link.lagg.0.lacp.lacp_strict_mode’. If it’s ‘1’ then set it to 0. My original intention was for this to default to 0, but apparently that didn’t happen. However, the fact that strict mode doesn’t seem to work at all for you might hint that your switch either isn’t configured correctly for LACP, or doesn’t actually support LACP at all. You might want to investigate that.

Scott

[Ben]

Hi Scott,

I had tried to set it in /etc/sysctl.conf but seems it didnt work. But
will I try again and report back.

The settings of the switch have not been changed and are set to LACP. It
worked before so I guess the switch should not be the problem. Maybe
some incompatibility between FreeBSD + igb-driver + switch (Juniper
EX3300-48T).

I will update you after setting the sysctl setting. It seems to be
"dynamic", I guess 0 reflects the index of LACP lagg devices. Can I
switch off the strict mode globally in /etc/sysctl.conf?

Thanks for your help.

Regards
Ben

> !DSPAM:1,52ef53f7888821422342440!

[Scott Long]

Hi,

Unfortunately, you can’t control the strict mode globally. My apologies for this mess, I’ll make sure that it’s fixed for FreeBSD 10.1. If the sysctl doesn’t help then maybe consider compiling a custom kernel with it defaulted to 0. You’ll need to open /sys/net/ieee802ad_lacp.c and look for the function lacp_attach(). You’ll see the strict_mode assign underneath that. I’ll also send you a patch in a few minutes. Until then, try enabling net.link.lagg.lacp.debug=1 and see if you’re receiving heartbeat PDU’s from your switch.

Scott

[Ben]

Hi,

I set strict mode to 0 but no use. I do receive PDU messages.

igb0: lacpdu transmit
actor=(...)
actor.state=4d<ACTIVITY,AGGREGATION,SYNC,DEFAULTED>
partner=(...)
partner.state=0
maxdelay=0

Thanks
Ben

> !DSPAM:1,52ef5c19888823082815771!

[Scott Long]

Did you set it to 0 via the sysctl? You might need to wait for several minutes if you set it after setting up the links.

Also, the message that you’re seeing is from your machine transmitting PDU packets. Are you seeing any "lacpdu receive” messages on the console?

Thanks,
Scott

[Ben]

Yes, via sysctl and /etc/sysctl.conf

I waited now roughly 20 minutes without touching it but no difference.

No, I only see these transmit messages, no receive.

Thanks
Ben

> !DSPAM:1,52ef6078888821231914487!

[Scott Long]

Please try the following patch:

Index: ieee8023ad_lacp.c
===================================================================
--- ieee8023ad_lacp.c (revision 261432)
+++ ieee8023ad_lacp.c (working copy)
@@ -192,6 +192,11 @@
SYSCTL_INT(_net_link_lagg_lacp, OID_AUTO, debug, CTLFLAG_RW | CTLFLAG_TUN,
&lacp_debug, 0, "Enable LACP debug logging (1=debug, 2=trace)");
TUNABLE_INT("net.link.lagg.lacp.debug", &lacp_debug);
+static int lacp_strict = 0;
+SYSCTL_INT(_net_link_lagg_lacp, OID_AUTO, lacp_strict_mode,
+ CTLFLAG_RW | CTLFLAG_TUN, &lacp_strict, 0,
+ "Enable LACP strict protocol compliance");
+TUNABLE_INT("net.link.lagg.lacp.lacp_strict_mode", &lacp_strict);

#define LACP_DPRINTF(a) if (lacp_debug & 0x01) { lacp_dprintf a ; }
#define LACP_TRACE(a) if (lacp_debug & 0x02) { lacp_dprintf(a,"%s\n",__func__); }
@@ -791,7 +796,7 @@

lsc->lsc_hashkey = arc4random();
lsc->lsc_active_aggregator = NULL;
- lsc->lsc_strict_mode = 1;
+ lsc->lsc_strict_mode = lacp_strict;
LACP_LOCK_INIT(lsc);
TAILQ_INIT(&lsc->lsc_aggregators);
LIST_INIT(&lsc->lsc_ports);

>>> !DSPAM:1,52ef5c19888823082815771!

[Scott Long]

Ok, please try the patch I emailed earlier. Since you’re not seeing any receive messages, it means that your switch isn’t generating any LACP heartbeats. The difference between FreeBSD 9.x and 10 is that in 9.x, it ran in “optimistic” mode, meaning that it didn’t rely on getting receive messages from the switch, and only took a channel down if the link state went down. In strict mode, it looks for the receive messages and only transitions to a full operational state if it gets them. So while I know it’s easy to point at the problem being FreeBSD 10, seeing as FreeBSD 9 worked for you, please check to make sure that your switch is set up correctly.

I authored the original change that went into FreeBSD 10, and I tried to make it so that strict_mode=0 would keep everything working as it did in 9. I guess that since you’re getting no receive messages from the switch at all that we need to disable strict mode on setup, not afterwards. Apply the patch and everything should work as it did in FreeBSD 9.

Scott

On Feb 3, 2014, at 2:29 AM, Ben <mailin...@niessen.ch> wrote:

> Yes, via sysctl and /etc/sysctl.conf
>
> I waited now roughly 20 minutes without touching it but no difference.
>
> No, I only see these transmit messages, no receive.
>

> Thanks
> Ben

>> !DSPAM:1,52ef6078888821231914487!

[Ben]

Thank you for your detailed explanation.

If I understand correctly the switch is probably not set up correctly,
right?

I will try to have it configured correctly first.

Thanks a lot for your help!

Regards
Ben

>>>>>>>> This is a screenshot of ifconfig -a after the upgrade to FreeBSD 10..0-RELEASE: http://tinypic.com/view.php?pic=28jvgpw&s=5#.Uu9PXT1dVPM

> !DSPAM:1,52ef6540888822133843295!

[Sami Halabi]

Hi,
Changing in/etc/sysctl.conf isnt enough unless you restart the machine.
Do in shell:
sysctl net.link.lagg.0.lacp.lacp_strict_mode=0

Sami
בתאריך 3 בפבר 2014 11:51, "Ben" <mailin...@niessen.ch> כתב:

>>>>>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@
>>>>>>>>> freebsd.org"

>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> freeb...@freebsd.org mailing list
>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net

>>>>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@
>>>>>>>> freebsd.org"

>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>> freeb...@freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net

>>>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@

>>>>>>> freebsd.org"
>>>>>>>
>>>>>> _______________________________________________
>>>>>> freeb...@freebsd.org mailing list
>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>> To unsubscribe, send any mail to "freebsd-net...@freebsd.org
>>>>>> "
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>> freeb...@freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>>>>>
>>>>
>>>> _______________________________________________
>> freeb...@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>>
>> !DSPAM:1,52ef6540888822133843295!
>>
>>
>>
> _______________________________________________
> freeb...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>

[Ben]

Hi,

Sorry for not making it clear. Sure I restarted the machine completely
and set it at runtime as well. Both made no difference.

Regards
Ben

> !DSPAM:1,52ef67e5888828188920253!

[Scott Long]

Hi,

If you can, please test the patch I sent and let me know the results. I’ll check it into FreeBSD 11 and 10 if it works for you.

Thanks,
Scott

>>>>>>>>> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>>>>>>>> _______________________________________________
>>>>>>>> freeb...@freebsd.org mailing list
>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>>> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> freeb...@freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>>>>>> _______________________________________________
>>>>>> freeb...@freebsd.org mailing list
>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> freeb...@freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>>>>
>> _______________________________________________
>> freeb...@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>>

>> !DSPAM:1,52ef6540888822133843295!

[Ben]

Hi,

It was Juniper's active/passive mode regarding LACP.

It was set to passive and worked as you described without sending any
packages. Now it was set to active and works perfectly again.

I couldn't try your patch easily as I didn't have the sources installed
(and obviously no network connection).

If the time allows I will try your patch anyway.

Thanks for your help!

Regards
Ben

>>>>>>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd..org"

> !DSPAM:1,52ef691c888821141715696!

[Kubilay Kocak]

On 3/02/2014 9:20 PM, Ben wrote:
> Hi,
>
> It was Juniper's active/passive mode regarding LACP.
>
> It was set to passive and worked as you described without sending any
> packages. Now it was set to active and works perfectly again.
>
> I couldn't try your patch easily as I didn't have the sources installed
> (and obviously no network connection).
>
> If the time allows I will try your patch anyway.

It would be *great* if you could do that Ben :) Having a successful
real-world test case will provide Scott the confidence to land a commit
and merge it back to stable/10 so that everyone can benefit as soon as
possible.

--
Koobs

[Ben]

>>>>>>>>>> freeb...@freebsd.org mailing list
>>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net

>>>>>>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd..org"

> !DSPAM:1,52ef691c888821141715696!
>
>

Hi,

I tried your patch and it works. The strict mode is now set to 0.

One thing I saw:
There is a message at the login prompt: igb0: Interface stopped
DISTRIBUTING, possible flapping
igb0 and igb1 are used for the lagg device.

I still get the following messages when I restart netif:
can't re-use a leaf (lacp_strict_mode)!
can't re-use a leaf (rx_test)!
can't re-use a leaf (tx_test)!

sysctl says strcit_mode is off at that time.

I hope this helps.

Best regards
Ben

[Scott Long]

On Feb 3, 2014, at 7:01 AM, Ben <mailin...@niessen.ch> wrote:

>
> Hi,
>

> I tried your patch and it works. The strict mode is now set to 0.
>
> One thing I saw:
> There is a message at the login prompt: igb0: Interface stopped DISTRIBUTING, possible flapping
> igb0 and igb1 are used for the lagg device.
>
> I still get the following messages when I restart netif:
> can't re-use a leaf (lacp_strict_mode)!
> can't re-use a leaf (rx_test)!
> can't re-use a leaf (tx_test)!
>
> sysctl says strcit_mode is off at that time.
>
> I hope this helps.
>

Thanks a lot for testing this. The ‘flapping’ message is intentional, it points out that something is wrong with heartbeat exchange with the switch. The “can’t re-use a leaf” messages point to bugs that I’ll fix shortly.

Scott

[Craig Duncan]

Apologies for replying to an old thread but I came across this via google.

I've been using NAS4Free for a while now and was tinkering with Link Aggregation on version 9.2 (sorry not sure what version FreeBSD this is based on) and decided to upgrade to version 10.0.0.2 (1906) which is based on
FreeBSD Revision: 199506
FreeBSD 10.2-RELEASE-p5 #0 r289200M: Tue Oct 13 00:54:46 CEST 2015

I didn't have problems setting things up on 9.2 but am having problems on 10.0.0.2

Basically I saved a config file on 9.2 and restored it after a clean install of 10.0.02 as was suggested to me.

The ip I was using seems to be consistant before I restore and I can access it via NAS4Free's web gui (although only 1 of the 2 nics appear to be active) but when I restore my config, both nics seem to be active and the same ip is kept but I can't access the web gui.

Unfortunately there are not many people that knowledgable re LAGG problems at the NAS4Free community and I came across this thread while searching:

"Initializing interface. Please wait...can't reuse a leaf (lacp_strict_mode)!"

which is what I've been getting.

So, can I ask if these LAGG problems are all sorted on this build of FreeBSD so I know whether to look for answers here or on NAS4Free?

Many thanks in advance
Craig.