Change of Behaviour Extreme Networks X690 for QSFP Ports

An unexpected change in behaviour was observed in terms of the interlink ports 57,65. The default beheaviour of the ports on previous software version: 22.6.1.4 was that being they are QSFP (40Gbit) ports that they operated at 40Gbit being that the QSFP DAC cable was 40Gbit, i.e. there was no specific port configuration for the speed or duplex, but it set to 40Gbit/Full.

Upon the upgrade to 32.7.2.19 this default configuration changed to “auto-configuration”, which then meant that during the upgrade the MLAG pair of links both went down (on the upgraded switch) causing a “split-brain MLAG” and the subsequent odd (loop-like) behaviours. The issue was not related to the subsequent enablement of SLPP which was a red-herring in terms of the issue.

Observed Behaviour

The behaviour that was observed was as follows:

1. The Switch A was upgraded to 32.7.2.19, and the rebooted.

2. Connected hosts continued to operate unaffected, because their connectivity was maintained via the surviving Switch B and MLAG configuration ensured traffic was sent only to the up and surviving Switch B.

3. Once Switch A had booted and returned to operation, issues with connectivity to both the switches’ inband IP addresses and connected hosts was observed. It was noticed that the two interlink ports (MLAG) peer ports (ports 57,65 in our case) were both down and the following was shown in the logs on the upgraded switch, which indicated a change in behaviour.

11/10/2025 13:26:11.81 <Noti:HAL.Port.OpticAutoNegEnblByDflt> Auto-Negotiation enabled by default for the transceiver module in port 57. Please verify that the peer has the same setting. 
11/10/2025 13:26:11.81 <Noti:HAL.Port.OpticAutoNegEnblByDflt> Auto-Negotiation enabled by default for the transceiver module in port 65. Please verify that the peer has the same setting.

The interlink ports did not come up, effectively meaning that the MLAG was in a “Split-Brain” state, traffic was not reaching the hosts or the switches’ in-band IP addresses as expected. Potentially setting the ports 57, 65 to hard-coded port speed (40G/Full) on the Switch A would have allowed it to match and therefore for the links to come up and the MLAG to return to a normal state, however this was not tested at the time.

4. A workaround was put into place by shutting down all the ports (apart from 57 and 65) on Switch B, this isolated switch B completely and removed the “split-brain MLAG” situation and returned traffic to normal, albeit with the loss of resilience for now.

5. The Switch B switch was then upgraded to the same version as switch A.

6. The interlink ports 57 and 65 were enabled on Switch B, now that both had the same default behaviour for the 40Gbit ports, i.e. auto-configure, the links both became live and the MLAG interlink/peering returned to normal operation. At this point all traffic was operating as expected.

7. The uplink port 49 in this case was then enabled, the MLAG completely formed, again no impact to network traffic.

8. Finally the edge ports on Switch B were re-enabled to return the switch pair to normal operation. No observed impacts to network traffic.

Conclusion

It was noticed that there is a support article: https://extreme-networks.my.site.com/ExtrArticleDetail?an=000117218 that explains the situation were found ourselves in.

There was a change of behaviour from 32.6.2 onwards which changed the behaviour of the ports to use auto-configuration for the QSFP ports by default, previously these used a 40G/Full by default being that they were QSFP ports, the change of behaviour essentially meant it was like having auto-configuration at one end and hard-set at the other which caused the links to come up.

https://supportdocs.extremenetworks.com/support/release-notes/product/extremexos-software

https://documentation.extremenetworks.com/release_notes/ExtremeXOS/32.6.2/GUID-F4F5039C-70B9-4548-8470-C40DD720C47B.shtml

Leave a comment