WN Blog 013 – Cisco Catalyst 9800-CL – Redundancy HA SSO (CLI and Deeper Dive)

Hey! And welcome to another C9800 blog 🙂 Recently, Matt has blogged about Redundancy configuration using GUI and covered HA/SSO operation here. Today, I wanted to do a deeper dive into the topic. Enjoy!

Lab Environment:

In my lab, both C9800-CL VMs will sit on a single ESXi box. In the real world, they would rather be distributed across separate VM hosts or even separate geographical locations. In any case, configuration would stay identical and same pre-reqs are still relevant. Most importantly, both vWLCs need access to the LAN on their mgmt. interface (that will also be used by APs to register to) and a separate ‘P2P’ L2 link to form & monitor HA and sync configuration.

See a full picture of my lab below:

And to make things simple, also see a simplified logical diagram of C9800-CL networking and interfaces mapping to the VMware vSwitches:

HA/SSO nitty-gritty stuff

• HA/SSO in C9800 works differently to how it works in Aironet, it’s more like a VSS now.
• Initially, both C9800-CL WLCs will require management IP address.
  • Note, that management IP is optional for a secondary WLC if you have a console access to it. Mgmt IP is no longer required to form a HA Pair
• Redundancy Management as we know it from AireOS (additional IP per WLC from the same subnet as mgmt.) doesn’t exist anymore and therefore Redundancy Port IP is no longer automatically derived from the Redundancy Management IP (it used to be 169.254. + last two octets of the Redundancy Management IP) – you have to manually specify “Redundancy Port” IP and elect one of the VM interfaces to be used as a Redundancy Port (RP)
• “Redundancy Port IP” is now called “Redundancy Local IP” (GUI) or “Local HA Interface IP” (CLI) <- same thing, just called differently in different places
  • Local HA Interface IP (I’ll call it just that as we’ll use CLI to form HA/SSO in this example) should be in a different subnet to a management interface. This subnet should not be routable.
• Both WLCs are using the same mgmt. IP to maintain AP CAPWAP during failover once HA/SSO is formed
• Two units are using a dedicated Redundancy Port (now called HA-Interface) to sync
• Role election happens at boot, uses priority (1-15, default is 1; higher priority is preferred) or lowest MAC if priority is the same; two controllers are supported, not more
  • It’s best practice to assign higher priority to the preferred active WLC
• Standby WLC continuously monitors Active via RP keepalives
• Gratuitous ARP is sent by a Secondary WLC that is taking over the Active role
• No preempt functionality is supported – failback, if needed, is manual
• Timers:
  • SSO Failure Detection: 50 ms
  • Reconciliation Time (Standby becoming Active): max 1020ms
• Reloading Active WLC via CLI: # reload will also reload a standby immediately
• Forming HA wipes out the config!!!
  • Always form a HA first, configure later
• WLC Interface used for HA will disappear from the interfaces list (sh ip int br etc. will not list it anymore)

HA/SSO Pre-reqs

• HA Pair can only be formed between two WLCs of the same form factor (C9800-CL VMs in this example)
• Both WLCs must be running same code version
• Max RP RTT = 80ms, min bandwidth = 60Mbps, minimum MTU = 1500
• Both WLCs are built, accessible, VMware networking is configured, WLCs have access to the LAN and separate L2 link for the HA is available between the WLCs
  • C9800-CL HA-Interface that we intend to use as a Redundancy Port L2 HA inter-vWLC link must be put into a separate, unused VLAN! Using existing VLAN/subnet, especially if it’s management VLAN/subnet, would theoretically work (I’ve tested it) but it will cause instability, duplicate ARP entries and ARP resolution issues. See below the ‘show logging’ output from my switch, where I had HA-Interfaces IP sitting in the mgmt. subnet:

• See full deployment guide of the C9800-CL for VMware ESXi in our previous blog here
• Prepare IP addresses and subnets; in this example we will use the following:
  • Active WLC
    • Chassis 1, priority 2
    • Mgmt. (VLAN 11): 10.10.11.35 /24
    • HA Local IP (VLAN 666): 10.10.66.35 /24 (VLAN 666
  • Backup WLC
    • Chassis 2, priority 1
    • Initial Mgmt (VLAN 11): 10.10.11.40 /24 (optional if have console access, won’t be used anymore once HA is formed)
    • Mgmt: (VLAN 11) 10.10.11.35 /24 (shared between HA WLCs once HA is formed)
    • HA Local IP (VLAN 666): 10.10.66.40 /24

HA/SSO Configuration

I feel CLI gives us better visibility into what is happening on both WLCs while forming HA Redundancy, therefore we will use CLI here. It is possible to use GUI too, but I am not a huge fan of “click, wait, pray and hope for the best” approach, where I can’t see what’s happening 🙂

CLI

• Active C9800-CL
  • # chassis ha-interface Gig 3 local-ip 10.10.66.35 /24 remote-ip  10.10.66.40
  • # chassis 1 priority 2
  • # wr
  • # reload

• Standby C9800-CL
  • # chassis ha-interface Gig 3 local-ip 10.10.66.40 /24 remote-ip  10.10.66.35
  • # chassis 1 renumber 2
  • # chassis 1 priority 1
  • # wr
  • # reload

GUI

Refer to our previous C9800-CL Redundancy blog written by Matt here

Note: Standby WLC would need to be accessible on its management IP address to use GUI for Redundancy configuration.

Tshoot & Validation

It is good and useful to know how to monitor, tshoot and validate HA Redundancy configuration. You can do it in both CLI (see below) and GUI (Monitoring > System > Redundancy; Dashboard > Slot: Active / Standby)

Chassis

# show chassis

• Validate Chassis# and Priority. Also see the HA state (“Ready” is the final and desired stage, you can also see “Initializing” etc.) and HA-Local IP of both C9800-CL VMs.

# show chassis ha-status local

• Check before the reboot while forming the HA
• Active should have higher priority, HA Local and Remote IPs and assigned interfaces are shown, together with Chassis# and Priority that will be assigned to this local WLC upon reboot (you might have changed those values and it can be slightly confusing, so it’s best to check after configuring and before rebooting_

Redundancy

# show redundancy

• Another useful show command, where we can validate uptime in current state, versions, modes and more!

Want to Know More?

Yeh, totally get it – me too 😉 I’ll drop some useful stuff that I came across and used at some point here that might help with configuration and troubleshooting.

Manual failback

• # redundancy force-switchover
• Can be very useful in scenarios like primary DC maintenance, where Backup C9800-CL takes over. Since there is no preempt mechanism in C9800 HA (former Active WLC won’t claim its original Active role back when it comes back online)

Clear HA Configuration

• # chassis clear
• # reload

This is how it looks in the CLI:

Note: it’s enough to clear chassis information on the active HA Pair WLC, despite the information that the ‘other’ (Backup) chassis will keep its HA configuration. I’ve tested it and it will not! Backup chassis will also reboot with clear redundancy config!

• Chassis clear is used to break the HA; be careful with this as you will end up with duplicate IPs! Both former Active and Passive WLCs will share identical configuration upon reboot!

Successful HA/SSO Formation

This is how successful HA  formation should look like in a console of both Active and Passive WLCs:

Successful HA/SSO Formation upon reboot once ha-interface was configured; Active (left) was elected Active due to higher priority

That’s it for now! It feels great to play around with new toys and discovering differences between old and new WLCs! Don’t hesitate to comment, add to the discussion or simply let the world know that you are enjoying beta-testing Cisco gen 1 products as much as we are 😉