Cisco

WN Blog 029 – Setting Up MPSK on a Cisco C9800 WLC

Hey!

Welcome to our latest blog, today we will be showing you how to configure a Cisco C9800 WLC to use the “MPSK” (Multi Pre Shared Key) feature.

Multi-PSK feature supports multiple PSKs simultaneously on a single SSID. You can use any of the configured PSKs to join the network. This is different from the Identity PSK (iPSK), wherein unique PSKs are created for individuals or groups of users on the same SSID.

Restrictions on Multi-PSK:

  • Central authentication is supported in local, flex, and fabric modes only.
  • In central authentication flex mode, the standalone AP allows client join with the highest priority PSK (priority 0 key). New clients that do not use the highest priority PSK are rejected during the standalone mode.
  • We can have up to 5 x MPSKs (0-4) with 0 being the highest priority.
  • Multi-PSK does not support local authentication.

For more information around central and local authentication check out our C9800 WLC series where we have covered nicely the difference between these methods of authentication.

Let’s take a look at my lab set up – I am running a virtual C9800-CL and it is on version 17.1.1s.

Now we will show you where you need to go to configure your WLAN to use the MPSK feature. First click on the “configuration” tab on the left and then we will select “WLANs” which is under the Tags & Profiles.

Then either pick one of your already created WLANs or create a new WLAN – in this example, I have called my SSID “Cisco-MPSK”. Once selected move on to the “security” tab inside of the WLAN.

Inside of here we will see many options, under “layer2” we want to make sure that we have the following WPA2 check boxes highlight below selected and then we can enable MPSK.

When you enable MPSK you will be able to start adding your MPSK’s and this is what the configuration box will look like. You will have the option to select the priority (remember 0 is the highest priority), which key format you want to use, if you would like the password type to be encrypted or unencrypted & then finally whatever you would like your MPSK to be.

When you have finished entering all of your MPSKs (remember we can have a maximum of 5) this is what your configuration will look like.

We can also verify that MPSK has been successfully enabled via the CLI.

If you run the following command – “#show wlan id 1” (You will need to make sure that you use the relevant WLAN ID number for your WLAN that you configured MPSK for)

Also we can run the “#show run wlan” command and we will be able to see here that MPSK is enabled on your WLAN and what keys are in use.

Ok, so now that we have verified that everything is configured as we are expecting it to be – let’s connect a couple of clients to make sure it works!

In this demo I used my two iPhones to connect to the MPSK and used a different MPSK for each iPhone and they connected first time without any issues at all. You can verify your connected clients in multiple ways on the Cisco C9800.

First of all right from the dashboard we can see my Top WLANs which is the profile Star9800 (this is the profile I configured the Cisco-MPSK) under and we can see that I have two clients connected. We can drill down further by clicking on this WLAN, or alternatively we can go to Monitoring > Wireless > Clients.

This is the view once we have gone to the Monitoring > Wireless > Clients view and we can see my two iPhones connected, their IP address’, which AP & SSID they are associated to & that they are in the run state.

If you click on one of the devices you can drill down even further to get more details & statistics for the client and this is what that view would look like.

Again, if you are a CLI kind of person we can verify the connected devices on CLI by using the following command – “#show wireless client summary”

There you go, that is just how simple & easy it is to set up MPSK on your Cisco C9800 WLC – something which is so powerful & useful for your wireless network. This lets you have accountability & improved security to your PSK wireless networks which I personally think is a much better option than using captive portals with open authentication or an SSID with a single PSK for all users.

I hope you enjoyed this blog and if you have your own Cisco C9800 AP at home or you use it currently in your production network – give setting up a MPSK SSID a go and let me know your feedback!

Much love, as always – WiFi Ninjas x

WN Blog 023 – Indoor RTLS with WiFi and BLE – Deep Dive

Hey! Welcome to our latest WiFi Ninjas Blog 🙂 We’ve been busy lately testing some available RTLS solutions from a few different major wireless players and fell absolutely in love with the topic to the point, where we shake from overexcitement when thinking or talking about it 😊

This summarises how we feel:

Excited Ninjas

Steve Jobs once said, “people don’t know what they want until you show it to them”. He was right.

20 years ago, WiFi was “a nice to have”. 18 years ago, paper maps were at peak of their popularity, until GPS receivers got small enough to be put in the handheld devices. Is anyone not using Google Maps on their mobile? 15 years ago, not many people felt they needed a wireless headset, until Bluetooth was introduced. 7 years ago, NFC was a niche. Today, most of us use contactless payments using cards, phones or watches.

Indoor wireless RTLS is at the early adoption stage. Technology is available, we just need to make it more interesting and rewarding to use.

Satellite-based location positioning services are not always practical indoors. We could offer wireless RTLS functionality based on WiFi, BLE or soon, possibly, Ultra Wide Band (UWB). Modern smartphones have both WiFi and BLE radios built in. Additionally, new iPhones support UWB. Wireless vendors leverage cell of origin, trilateration, triangulation, angle of arrival, proximity and more with WiFi, BLE, UWB, GPS, NFC, mobile networks or any combination of those methods and technologies, offering different levels of accuracy and functionality. RTLS market is growing fast – according to Bluetooth.com, 1.7 billion devices will use indoor BLE RTLS by 2023, translating to even 500% increase in BLE RTLS implementations in some verticals!

So, what do we need to see the RTLS market explosion? We’ll be speculating, but our hearts tell us that indoor RTLS needs solve real challenges like indoor wayfinding or call for help and simply be more interesting not only for the business’ IT and marketing, but for everyone. We also need more skilled wireless professionals willing to embrace broader wireless technologies and emerging use-cases, like BLE and RTLS, on top of WiFi design, security and analysis skills, while not forgetting about programmability elements.

But enough of the babbling, let’s cut straight to the juicy content! Here is what we will cover in this blog:

Blog Structure:

  • Theory
    • RTLS Technologies
      • WiFi
      • BLE
    • RTLS Tracking Methods
      • Cell of Origin
      • WiFi Trilateration
      • WiFi Angle of Arrival
      • Mist vBLE Arrays and Probability Surfaces
    • RTLS Functionality
      • Presence & Analytics
      • Location
      • Engagement & Actions
    • RTLS RF Design
      • Design Tips
      • Examples
  • Demo Time!
    • Test Environment
    • WiFi Trilateration
    • WiFi Hyperlocation
    • Mist BLE Arrays
  • Location API
    • Challenge 1 (Meraki Location API): Do You Need RTLS RF Design for WiFi Location & Presence Analytics?
    • Challenge 2 (Cisco CMX Location API): Can You Leverage Enterprise Messaging Solutions to Benefit from Indoor RTLS?
  • Gotchas

Theory

RTLS Technologies

WiFi

  • Most popular tracking technology, as it also provides access to the network.
  • Can offer different levels of location accuracy, depending on the tracking method used (cell of origin, Trilateration, AoA, etc.).
  • Multiple element antenna arrays can substantially improve location accuracy – Cisco Hyperlocation is a great example.
  • Can be used to track associated and unassociated probing clients.
  • Associated is proffered, since stations are chattier (with their screen on at least).
  • Normally, WiFi based RTLS is based on RSSI/Location information from probe requests only, as probing client normally sends requests on multiple channels, and is therefore seen by multiple APs. This, however, results in very infrequent location calculation ranging from 10 seconds to 5 minutes (according to Cisco) but our own tests shown that modern unassociated stations don’t probe at all. Note, that happy (high RSSI/SNR) associated client probably won’t probe at all, and therefore its location could not be calculated with better accuracy than ‘cell of origin’ (see below) until its re-associating (roaming). Cisco has addressed this challenge with a feature called ‘FastLocate’. It uses a dedicated built in antenna array (4800) or a halo module (3600 / 3700) to scan multiple channels and get RSSI values from clients’ data packets across multiple channels without sacrificing performance of clients serving radio by going off-channel to do just that.
  • To track unassociated station, two conditions must be met: station must not use MAC randomisation (they normally do) and it must be probing. Not all unassociated stations are probing.
  • Client apps are optional and can be used to increase tracking accuracy, location calculation frequency and add engage or actions element. Note, that using mobile SDK in WiFi-based RTLS is challenging, as the application normally has no way of knowing the MAC address of the device it sits on.
  • On top of location analytics, WiFi is normally used to provide presence analytics (more info below).

BLE/vBLE

  • BLE relies on physical beacons, often battery operated; vBLE moves beacon role from BLE beacons to APs .
  • BLE is often used to offer proximity-triggered actions and vBLE can be used for location analytics on top or instead of WiFi
  • Client app is required for fast (sub-second with Mist) and accurate (1-3 metres) BLE/vBLE. Station listens to the BLE transmissions coming from the BLE AP (BLE APs are transmitting). Mobiles listens to all (directional with Mist) beams from all BLE APs in the area and sends RSSI values with beams ID to the server/cloud, where location engine will calculate station XY coordinates and send them to the station. Note, that mobile station can send that information using mobile network; device doesn’t have to be associated using WiFi for BLE RTLS to work.
  • Clients with no apps will be treated / located as BLE tags / assets. In this scenario, client is sending BLE transmissions to BLE APs (BLE APs are receiving). Mist uses its directional arrays to pinpoint asset tag transmission location with mind-blowing accuracy, all other vendors will typically rely on tag proximity to BLE radio built into the AP / anchor.
  • More BLE beams heard = better accuracy.
  • More element antenna arrays can substantially improve location accuracy – Mist BLE Arrays are a great example.
  • BLE seems to be very well suited for RTLS, as it doesn’t travel as far as WiFi, therefore offering theoretically better accuracy than WiFi. Note that both BLE (2.4GHz) and WiFi (2.4 or 5GHz) are using the same frequency (actually WiFi can even sit on a higher frequency with 5GHz, and therefore theoretically not travel as far as 2.4GHz) but BLE (Bluetooth LOW Energy) uses significantly lower power levels to operate.
  • XY coordinates are calculated for vBLE and are typically not calculated for BLE (proximity used instead, but there are exceptions to that rule!).

RTLS Tracking Methods

Cell of Origin

  • Simplest location technique.
  • Typically leveraging location of AP that clients are connected to but might be the AP that sees the probing or associated client with strongest RSSI.
  • Great for simple zone-wide accuracy.
  • Requires at least 1 AP per zone and careful RF design.
  • XY coordinates are not calculated.

WiFi Trilateration

  • Distance (lateration) based location technique using RSSI in 802.11, measured by either STA or AP/Sensor.
  • Adds more accuracy within a zone (tested with Meraki and Cisco) – 2-3m labbed (in perfect environment), 5-7m marketed, 7-10m real.
  • Real accuracy is lower than labbed and marketed, as it’s difficult to have LOS between 3x AP and STA everywhere. Also, cross floor leakage, atriums, walls and signal deterioration – all affect accuracy.
  • Note, that we normally use Trilateration in WiFi RTLS and it’s often confused with Triangulation.
  • Trilateration: requires 3 APs with known distance between them; uses RSSI (distance, lateration) to calculate intersection (client XY coordinates) between three circles (as shown above).
  • Triangulation: requires at least 2 APs with known distance between them; uses this baseline and multiple-element antenna arrays (that we typically don’t have in WiFi APs, with Hyperlocation being exception here) to calculate arriving signal angles to find XY coordinates of the client (as shown below).
  • XY coordinates are calculated.

WiFi Angle of Arrival

  • AoA (angulation) location technique using angle of incidence at which STA signals arrive at the receiving sensors.
  • Requires at least 2 APs / sensors / modules (at least 3 recommended for better accuracy).
  • Note, that AoA with three APs is in fact a tri-angulation.
  • Requires multiple element antenna arrays or antenna mechanical agility.
  • Cisco Hyperlocation uses a mix of WiFi Trilateration (RSSI), WiFi AoA and BLE
  • Cisco 3600/3700 Hyperlocation module has 32-element antenna array
  • Cisco 4800 built-in Hyperlocation has 16-element antenna array
  • According to Cisco, Hyperlocation module used with older APs and 4800 built-in array offers same levels of accuracy
  • AoA is more accurate than Trilateration (tested with Cisco Hyperlocation) – 1-3m labbed, 1-3m marketed, 1-5m real
  • Real accuracy is lower than labbed and marketed for similar reasons as discussed for Trilateration. Additionally, AoA requires extremely careful mounting and calibrating APs positions in maps services (height, azimuth).
  • XY coordinates are calculated.

Mist vBLE Array and Probability Surfaces

•aoo 
BEAMS 1 HEAR 
CLOUD
  • Unique to Mist.
  • Every Mist AP has 16 Directional Antennae Bluetooth Array – 8 reflectors and 8 directional antennas.
3-
  • Uses BLE concept, where Mist SDK on the mobile device is listening the beacons from the beams and sends the RSSI and device sensor information back to the Mist cloud through either WiFi or cellular. Mist also supports assets tracking.
  • Mist is not using standard Trilateration, but ‘Probability Surfaces’, where combination of listening to directed AP Beams and machine learning constantly evaluating Path Loss Formula (PLF) per device to calculate station location.
  • Extremely good accuracy: 1-2m labbed, 1-3m marketed, 1-3m real (mind blasting).
  • XY coordinates are calculated.

RTLS Functionality

Presence & Analytics

  • Typically relies on WiFi
  • Provides network level stats, such as:
    • Current visitors: devices count, dwell time, gender and age split, device types present
    • Avg. visit distribution: time of day, day of week, new vs repeat, duration, gender, age
    • Bounce rate (stayed vs bounced; enter and stay for longer than 3 min)
    • Conversion rate (converted vs passed; clients that passed the venue but didn’t enter)
    • Visitors engagement ratio (stayed connected over specified time)
  • We can leverage social media, mobile apps or splashpage forms to onboard WiFi clients and grab users’ details (be careful with GDPR!)

Location

  • Typically relies on WiFi, BLE/vBLE, GPS, Mobile, UWB, Ultrasounds
  • Provides zone/location level insight, such as:
    • Zone analytics & zone paths
    • Location on the map (dashboard or client app – blue dot)
  • Some location features can be leveraged directly through specific web user interfaces (CMX, Purple, Mist or Meraki Dashboards) but it’s generally more powerful and useful to get, manipulate, filter and visualise location data using API – output can be crafted to any business needs. Typically, the following attributes can be grabbed (or subscribed to if you’re using Mist Webhooks) via location API:
    • Client XY coordinates
    • Zone entry and exit events
    • Virtual beacon (Mist), beacon proximity (everyone else)
    • Raw tracked client’s data: RSSI, time, MAC, etc.
  • Zone analytics importance
    • Zone analytics is extremely valuable for businesses, especially retail, to understand impact of their actions (new displays, stands, brands, promotions, etc.) on paths users are taking and stats for each zone (dwell time, count, etc.)
  • CMX GUI – Zone Paths example:
A screenshot of a video game

Description automatically generated
  • Purple GUI Zone – Paths example:

Engagement & Actions

  • Uses Presence, Analytics and Location insight to create personalised user experience
  • Example 1:
    • Returning user that has opted in and has previously bought a pizza (tagged with ‘pizza lover’ tag) entering 1st Floor dining area after 3.15pm will get a 50% off offer delivered via App Push Notification (This would also require some CRM integration)
  • Example 2:
    • When user is in a proximity to a BLE beacon or a vBLE area, open wifininjas.net website in a loyalty app integrated browser
  • Example 3:
    • Zone A (sports cars) will use different Captive Portal than Zone B (donuts)
  • Example 4:
    • User uses app for indoor wayfinding – where am I and how can I find client X reception and how to get there?
  • Example 5:
    • User requires assistance in the shop (looking for a certain size of shoes to try); user can use the mobile app to ‘call for help’ and grab attention of staff; staff knows where to find a client calling for assistance as they also use location-aware mobile app

RTLS RF Design

Design Tips

  • APs should be located around zone perimeters to create a convex hull
  • Each client should be within convex hull of at least 3 APs with solid RSSI (-65dBm is OK)
    • It’s ideal for all tracking methods, even though AoA and Mist BLE require just 2 APs to work; still, more APs give better accuracy
  • Ideally, use dedicated radio or module for RTLS to maximise location scanning performance and minimise performance impact to client-serving radios
  • Ensure LOS is maintained between APs and clients (AP behind ceiling tiles is a no-no)
  • Don’t mount APs too high! Not more than 4.5m is OK
  • Validate secondary and tertiary signal strength in your favourite survey tool
  • Cisco specific:
    • Enter thick walls into Prime and ‘Enable OW (Outer Walls) Location’ in CMX to use it for calculation
    • Always use FastLocate functionality (leverage client data packets and probes to calculate location)

Examples

Note, that number of APs has tripled! Generally speaking, if the design is good for RTLS, it’s also good for any other use case (data, voice, most high-density scenarios, etc.). Sometimes some WiFi radios need to be put into non-client-serving mode as too much WiFi can be damaging to the performance of the network (topic for a different discussion).

Demo Time!

Test Environment

  • All APs were positioned quite nicely, LOS with a client maintained everywhere within a convex hull making it slightly unrealistically great (normally possible in lab environment only).
  • Laser tool was used to measure height of the APs.

WiFi Trilateration

Components

  • Tested with traditional Cisco and Cisco Meraki
  • We have deemed Meraki Dashboard unusable for RTLS (shows only point of association and overlays all clients from all floors on every floor map) so we’ve also used Purple as an RTLS Dashboard overlay with Meraki testing
  • Prime (or DNAC) is needed to create maps, place APs, set their height, azimuth, specify zones, inclusion & exclusion areas, walls, rails (wayfinding paths)
  • Theoretically CMX is not required if Hyperlocation is not in use and we could connect AireOS or C9800 WLC directly to the DNAS. Unfortunately, we couldn’t do it as:
    • Direct connection requires DigiCert CA Root Certificate that we don’t have on the WLC
    • C9800 code version 16.12.2 is required, but 16.12.1 was newest publicly available at time of testing

Accuracy

  • Theoretical expected accuracy of 5-7m
  • Better than expected accuracy when inside the convex hull (2.55m average; 2.69m 90% error distance)
  • Still good accuracy when on the edge of the convex hull (2.57m average; 3.38m 90% error distance)
  • Location Computation Frequency of 11-15 seconds (with screen on, associated and active device) is not enough to offer blue dot experience but it’s still good for basic RTLS functionality like simple, static wayfinding or zone analytics with zones sized accordingly to solution accuracy

Cisco Hyperlocation

Components

  • Prime or DNAC still needed for maps
  • Overall complicated setup, prone to user errors and bugs
  • CMX on-prem required to do all data crunching – it’s quite a lot of data to go through!
  • DNAS can be used to access presence, analytics and location data in the cloud if required
  • Note, that at the time of testing DNAS was lagging behind CMX and was generally slower and less accurate than CMX!
  • DNAS can also be used for many more things, like integration with 3rd party tools or Open Roaming integration (amazing idea that we feel will change the way we use public WiFi – let’s leave it for a future blog 😉)

Accuracy

  • Theoretical expected accuracy of 1-3m
  • Totally brain smashing accuracy inside the convex hull (0.73m average; 0.73m 90% error distance) for a non-moving client
  • Good accuracy at the edge of the convex hull (1.78m average; 2.03m 90% error distance) for a non-moving client
  • Note, that accuracy of mobile clients will depend on their speed of movement and since location computation frequency is 2.3-2.4 seconds at best, the distance travelled within that timeframe will be typically added to the calculated location error distance

Demo

Let’s look at the video showing CMX app, CMX Dashboard and Matt walking around my house 😉

  • Accuracy is generally really good!
  • Blue dot location update (location calculation frequency) of 2-3 seconds is not bad and should be enough for most indoor RTLS use cases.
  • We didn’t draw rails (wayfinding paths) in Prime so what we’ve seen here is raw XY calculations displayed on the map. Rails around the path Matt’s taken would snap him into it, making blue dot appear more accurate.
  • CMX has great zone path visualisations built in and it’s generally a rarity (Purple also has it built in and it’s as good; note that not everyone will leverage GUI-based zone paths analytics since we can use API and visualise it ourselves and integrate it with existing tools but it requires clearly defined requirements and solid skills and effort).

Mist: vBLE Array

Components

  • Yup, that’s it 😉 Class leading simplicity. We’ve had a lab running by lunchtime and never had a chance to play with Mist before!
  • All maps management, wayfinding paths, beacons and zones done easily in the Dash.

Accuracy

  • Less than 1m inside the convex hull.
  • Reliable sub second Location Computation Frequency when used with an app, making it difficult for other major WiFi vendors to compete with.

Demo

The video screams million words! Let’s take a look 😊

  • Absolutely mind-bending accuracy with quickest location computation frequency we’ve seen to date in indoor RTLS space!
  • It is expected to still use WiFi to get network-wide presence and analytics stats, despite using BLE for location.
  • Mobile app is required for the proper BLE solution to work (might not be a bad thing, since mobile app is required anyway to get the most of any indoor RTLS solution, despite the technology used).
  • Mist has no zone path visualisations built in (we only have basic zone analytics) – it’s expected to leverage API to access / visualise this data.

Accuracy Summary

Below is a quick summary showing results of our tests for different indoor RTLS solutions!

Location API integration with existing infrastructure – two examples!

Meraki API – Cell of Origin Example

Background

We have recently worked on project for a beautiful, listed retail store in London, where following a very successful design and implementation of Meraki data WiFi network in the store, retail RTLS topic has emerged. With RF design crafted for data, we have faced a very real challenge – would the existing placement, type and number of APs be useful for indoor Real Time Location Services with a zone-level accuracy?What would be the challenges, limitations and reasonable expectations in terms of RTLS usage and accuracy? Could Meraki API help achieve goals set by the business?

Challenges

RTLS was not in the original scope and, combined with a store physical environment, we have faced some challenges:

  • RF Design: the network was designed with data in mind. We didn’t have APs placed on the outside coverage zones’ perimeters, had just 1 or 2 APs per zone in most areas and proved with a detailed WiFi survey/assessment that zone accuracy leveraging WiFi trilateration should not be expected.
  • Physical Environment: the building was listed, APs/antennas mounting options limited, walls and ceiling mostly wooden, with very low attenuation. All three sections (east, west, central) had big atriums contributing to cross-floor RF leakage strongly. Building had 6 levels.
  • Vendor Choice: we had Cisco Meraki and Purple already in place. Purple leverages Meraki API and uses Meraki pre-calculated XY coordinates (this can’t be changed) to display users’ location on the map. With data design, we didn’t have enough APs per zone to rely on the Meraki XY coordinates. Trilateration requires min. 3 APs in a zone to work reliably. Normally, we would have just one best AP (strongest RSSI) in every zone, with additional two APs needed for trilateration being located in adjacent zones or even a floor or two away! This has contributed to wildly inaccurate location readings in Purple (approx. 22m 90% error distance, often on a wrong floor) and could not be relied on. Note, that network-wide WiFi-based Presence & Analytics stats were perfectly reliable.

Floor diagrams below show the difference in RF Design for Data (what we have) vs RF Design for RTLS (what we would need for WiFi Trilateration to work):

Expectations

Based on the additional very detailed survey (more than 100 test locations across multiple floors with 3 different device types: flagship Windows 10 laptop, Android phone and iPhone), where we temporarily installed several new APs on tripods, we concluded that achieving satisfactory and reliable zone-location accuracy  (10m 90% error distance) using WiFi trilateration would require to approximately double the number of APs. The business has decided not to install additional APs and knowing that a mixture of Meraki and Purple could not offer reliable location insights and that WiFi trilateration would not be practical to use at all (because of the design, regardless of a vendor), we looked for alternative solutions and new success criteria were defined:

  • Location analytics with zone accuracy was the new goal; access to zone-based stats & clients’ paths deemed critical.
  • Zone sizes were increased; sometimes a zone would be as big as a 30x20m open space, sometimes as small as 8x8m room.
  • Still use Purple for presence & analytics, guest splash page & social media integration.

Solution

We have gathered some facts: data & voice performance over WiFi, capacity, coverage and roaming were all rock solid across all WLANs (MAB/CWA for guests and EAP-TLS machine auth for corp) and for all test devices (company issued laptops, major OSes, newer and older phones). High density areas, like ground floor entrance hall, could get very busy during holiday periods (200+ associated devices in a small 15x20m area) and were covered by three high-gain dual-band sector antennas. 5GHz was almost free of channel contention with careful APs placement around atriums and use of properly tweaked RF Profiles: 20MHz channels width, limited max and min Tx power, disabled data rates of 11Mbps and lower. 2.4GHz was tweaked even more, with several APs having their radios off and max Tx power set to the level, where RSSI was generally lower on 2.4GHz than on 5GHz throughout the store and only OFDM rates were allowed. 2.4GHz was still considered best effort. Presence & analytics with Purple was spotless.

Switching focus to location zone analytics, we have confirmed with a detailed post-deployment survey, that our test clients were reliably associating with APs installed in the zones they were in. WiFi scans from unassociated devices taken in multiple spots in each zone revealed that we could use strongest RSSI reading from one best AP to accurately pinpoint the probing or associated device to the zone it was in. We concluded that we could potentially leverage two attributes to correlate WiFi devices with their current zones:

  • Associated devices only: use MAC address of the AP the device is associated with.
  • Associated OR unassociated probing devices: use MAC address of the AP that reports probing or associated device with strongest RSSI.

Two things to note:

  • WiFi MAC randomisation makes it impossible to track unassociated devices that use it.
  • New mobile platforms battery saving modes make those devices very quiet – they often won’t probe at all.

To further simplify our zone analytics calculations, we have decided to use just one attribute moving forward – MAC address of the AP that reports the probing or associated devices with strongest RSSI.

We have discovered that it is possible to get the RSSI attribute for every client seen by all near APs with Meraki API location data. Readings are provided every minute and contain RSSI values covering last 60 seconds. While it’s not very fast and could not be used for a blue dot experience, it is enough for our use case – historical view of zone paths and analytics.

Using strongest RSSI proved to be 95% accurate across entire building and 100+ test spots when correlating user (reporting AP) location with a zone.

Raw Data from Meraki API – note ap_mac, seen_time, client_mac and RSSI

As we know exactly which zone all the APs are installed on, we could easily generate reports showing clients paths with zone accuracy, time spent in each zone, number of clients per zone, etc. without relying on wrongly pre-calculated Meraki XY coordinates.

To automate the zone paths visualisation and zone analytics, client’s software development team has created a tool to do just that. At this point, imagination was the limit.

Conclusion

Sometimes we are limited by a solution functionality when trying to meet client’s requirements. In our example, Purple could only use Meraki pre-calculated XY coordinates and it was not practical or accurate enough (even for a zone-wide accuracy) to use with a data / voice RF design. Let’s answer our initial question. Do You Need RTLS RF Design for WiFi Location, Presence & Analytics? Short answer is “no” for presence & analytics and “it depends” for location. Presence & analytics provide us with network-wide stats, without the location awareness and therefore will not require RTLS design. Location, however, requires very careful RF design to provide different levels of accuracy. With a solid data RF design, where association and roaming trends are reliably predictable, we could expect solid zone-wide accuracy. Anything more accurate (trilateration, Hyperlocation, vBLE) would require more APs, detailed survey, RTLS RF design, careful placement of APs/antennas and pedantic maps services configuration with spotless APs positions, height and azimuth set.

Cisco CMX API – Location Analytics integration with WebEx Teams Chatbot Example

Background

At Natilik, we’re currently in the process of upgrading our showcase with Cisco DNA. The plan is to have full SDA Fabric, C9800-based Fabric Wireless, proper RF design with Cisco 4800 APs, CMX and DNAS in one part of the office and a mixture of AP43 with BT11 in the other part. We’d love to not only showcase Cisco Hyperlocation and Mist BLE Arrays in action, but also leverage the tech to offer indoor wayfinding with turn by turn navigation and onboard guests using Hotspot 2.0 (called Open Roaming by Cisco) for our visitors WiFi.

As of today, we don’t have the new showcase fully running yet, so all we have to play with is ‘standard’ corporate WiFi design for Voice and Data.

The above shows one out of the two floors we occupy. With current design, all we can use is Cell of Origin and, around reception, WiFi Trilateration. And that’s it. Location Calculation Frequency of 11-15 seconds is not enough to offer blue dot experience, but we thought it could still be good enough to solve some wayfinding challenges!

Challenge

We have Cisco AireOS Wireless with CMX, Cisco WebEx Teams and everyone has a corporate phone enrolled with Meraki MDM.

We would love to use the kit we have today to locate a colleague, zone or a meeting room by asking WebEx Teams chatbot about the location of person/location we’re after without the need to install the mobile application, as we can’t have a turn-by-turn indoor navigation just yet anyway.

Let’s imagine that Mac wants to find Matt (yeah, we don’t hold our hands all the time and sometimes attend different meetings on different floors, lol). How do I do it? How would WebEx Teams chatbot know about my or Matt’s location? How could it show or tell me how to find him? Is it really that helpful with approximately 10-12m 90% error distance accuracy? Can we still draw an indoor map showing where should Mac go to find Matt?

Solution

First, we had to set realistic expectations. We shouldn’t expect the solution to pinpoint user to his or her desk knowing, that calculated user location has 10-12m accuracy. Instead, we have decided to use zones that are big enough to account for that location accuracy.

Here are the zones we’ve created, ensuring that each zone has at least one AP. We’ve also considered expected AP association ensuring it all makes sense:

Now, we need to figure out what happens when Mac is asking WebEx Teams chatbot about Matt’s location.

Wait, what is a chatbot? Good question! Most modern enterprise messaging solutions allow to create custom chatbots with custom functionality. We are fortunate enough to have a proper coding nerd in our ranks, Darren, that finds coding chatbots as easy as you probably find putting your socks on. Darren has created NatBot (I wonder where the name comes from!), that can translate natural language with certain key words to specific actions.

To make it all fly, we’ll leverage Cisco CMX Location API, Cisco Meraki MDM API and a cloud map service (i.e. Mapwize) to visualise the results.

Finally, let’s look at detailed steps that are happening in the background when Mac is looking for Matt 😊

  • Mac Deryng asks NatBot via WebEx Teams: “Where is Matt Starling”?
  • NatBot notes who is asking and who is he/she looking for
    • Mac Deryng is asking
    • He’s looking for Matt Starling
  • Find correlation between names and corporate iPhones MAC addresses
    • NatBot leverages Meraki MDM API to find MAC address of Mac’s phone (MDM returns MAC address)
    • NatBot leverages Meraki MDM API to find MAC address of Matt’s phone (MDM returns MAC address)
  • Find location of MAC address on the map and correlate it with a zone overlaid on the map
    • NatBot leverages Cisco CMX Location API to find location / zone that Mac is in (CMX returns zone name)
    • NatBot leverages Cisco CMX Location API to find location / zone that Matt is in (CMX returns zone name)
  • Show results for Mac
    • NatBot displays zone name that Matt is in and displays a map of the floor, highlighting that zone
    • Additionally, NatBot displays a ‘map’ button
  • Mac knows which zone Matt is in, but he’s not sure how to get there; Mac clicks on the ‘map’ button in his WebEx Teams
    • NatBot builds a URL using specific Mapwaze syntax that contains start zone (Mac) and target zone (Matt) and opens that URL
    • Browser displays Mapwaze service, that uses pre-created zones, lifts, staircases and paths and shows exactly how to get from start zone to target zone

Sounds complicated? Watching this short video should clear things up!

Sky is the limit

This is just the tip of the iceberg and we have some more ideas about what to do next, once the new showcase is in! 😊 Here is the list:

  • Add integration with voice assistants – why type when can just ask a question?
    • Mac: “Alexa, where is Matt Starling?”
    • Alexa: “He’s in the toilet on the 1st floor. Take a lift or stairs, go right, and right again after 10 metres. You can also see the map on your screen”
  • Add integration with calendar
    • Mac: “Alexa, what’s my next meeting?”
    • Alexa: “Your next meeting is recording podcast with Matt. It’s in Fiennes meeting room on the 3rd floor. You’re also on the third floor. Walk out of the back office and take first left and look for second door on the right. You can also see the map on your screen”
  • Add full wayfinding functionality leveraging mobile SDK
    • Support both Cisco Hyperlocation and Mist BLE
    • Offer full turn-by-turn indoor wayfinding
  • Leverage mobile sensors to enhance the blue dot experience
    • Use phone accelerometer and compass to make the experience smoother
  • Offload indoor location with Hyperlocation to GPS and 5G
    • Why limit yourself to just inside?

Gotchas 

Lastly, here is some stuff that we think might save you some ball ache 😊

  • Location Computation Frequency for your mobile device will be affected when your screen is off or when your phone goes to sleep
    • Screen off (WiFi-based RTLS): typically doubles the time needed to calculate device location; normally your mobile will go to sleep after several seconds with screen off unless some apps running in the background require connectivity
    • Sleep (all methods): typically your device can’t be tracked when asleep; if apps are used, they would have to be excluded from battery saving modes for tracking to happen
  • NTP
    • Always use NTP – AoA won’t work when AP, WLC or CMX have even slight time differences
  • Components Compatibility
    • Always check compatibility matrix!
    • Newest software across the board (CMX, Prime, WLC) does not mean it will work (we’ve learned it the hard way)
  • APs Mounting
    • Mount APs carefully, especially if multi-element arrays are in use; make sure they’re level!
  • Maps Services Fine-Tuning
    • Set APs exact locations, height and azimuth in maps services! It MUST be spotless for the solution to work
  • Don’t mix Hyperlocation with non-Hyperlocation APs
    • Device associated with a non-Hyperlocation AP on will always be shown as ‘RSSI’ (and not ‘AoA’) in CMX, even if there are Hyperlocation APs nearby
  • Associate WiFi Clients
    • Modern devices won’t probe when not associated (with WiFi on), so tracking unassociated devices, in most cases, provide very little value (and even less with WiFi MAC randomisation)
  • Use Mobile Apps
    • Optional for WiFi, but can increase accuracy and sampling frequency; mandatory for BLE
  • Add C9800 to CMX as ‘Unified WLC’ using SSH, as opposed to ‘WLC’ using SNMP 

You’ve made it! This blog has almost 6000 words. Well done! 😊

With tons of love,

WiFi Ninjas x

WN Podcast 026 – RTLS – Part 2 – Real World Testing

Welcome to our new WiFi Ninjas Podcast episode!

This is the continuation of the RTLS discussion, focusing on real-world tests and demos!

Three things have changed since we recorded the podcast:

  • We’ve confirmed that you can draw wayfinding paths in Prime, called ‘Rails’ in a Cisco world!
  • We’ve discussed Mist presence and analytics network-wide stats that you can include in a report; new functionality has just been added, where you can see all that stats live! Big thing from Mist, thank you guys 🙂
  • We are discussing WiFi Trilateration (lateration, so distance based, without any special AP requirements – internal omni is OK) RTLS and NOT WiFi Triangulation (that is angle based and requires multiple antenna arrays or mechanically agile antennas)

With tons of love x,

WiFi Ninjas

WN Podcast 025 – RTLS – Part 1 – The Theory

Welcome to our new WiFi Ninjas Podcast episode!

Today we kick of the RTLS discussion, starting with a theory! We’ll cover how different technologies like WiFi or BLE can make RTLS work and what the RTLS really is.

We’ll discuss RTLS functional blocks like network wide presence stats, location aware functions like blue dot and zone location analysis and actions and engagement.

 It’s a vast topic, and we’ll release a blog about it shortly, so this show notes are rather short! Stay tuned 🙂

With tons of love x,

WiFi Ninjas

WN Blog 017 – Cisco Catalyst 9800 – Local Web Auth Configuration Guide

Hey!

Welcome to another one of our Cisco C9800 configuration blogs!

This time we will be covering Local Web Authentication (LWA), where guest sessions are managed by the WLC itself.

We can authenticate against RADIUS, TACACS, LDAP or local WLC Guest Users database. In this guide we will use local WLC Guest Users.

As I was preparing for deployment for a customer that would be using 2 x foreign C9800CLs in HA SSO & 2 x anchor C9800CLs in HA SSO I had my lab set up in this configuration also. In this scenario, the WLAN that I will be using is being guest anchored via a tunnel from the foreign WLC to the Anchor.

This meant that all the configuration you see below had to be replicated across both the foreign and the anchor WLCs.

We will include the steps we used from the official Cisco config guide but will add screenshots from our lab WLCs to hopefully make it a bit easier to follow.

Official Cisco guide:

https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_wl_16_10_cg/wireless-web-authentication.html

Below are the steps to configure LWA.

1. Configuring AAA Authentication (GUI)

Procedure

Step 1 Choose Configuration Security AAA.
Step 2 In the Authentication section, click Add.
Step 3 In the Quick Setup: AAA Authentication window that is displayed, enter a name for your method list.
Step 4 Choose the type of authentication you want to perform before allowing access to the network, in the Type drop-down list.
Step 5 Choose if you want to assign a group of servers as your access server, or if you want to use a local server to authenticate access, from the Group Type drop-down list.
Step 6 To configure a local server to act as a fallback method when servers in the group are unavailable, check the Fallback to local checkbox.
Step 7 Choose the server groups you want to use to authenticate access to your network, from the Available Server Groups list and click > icon to move them to the Assigned Server Groups list.
Step 8 Click Save & Apply to Device.

As we are using local WLC Guest Users database to authenticate against, we will specify ‘local‘ Group type for ‘login‘.

AAA Config

2. Creating Parameter Maps

Procedure

1 Choose Configuration Security Web Auth.
2 On the Web Auth page, click Add.
3 In the Create Web Auth Parameter window that is displayed, enter a name for the parameter map.
4 In the Maximum HTTP Connections field, enter the maximum number of HTTP connections that you want to allow.
5 In the Init-State Timeout field, enter the time after which the init state timer should expire due to the user’s failure to enter valid credentials on the login page.
6 Choose the type of Web Auth parameter.
7 Click Apply to Device.
8 On the Web Auth page, click the name of the parameter map.
9 In the Edit WebAuth Parameter window that is displayed, choose the required Banner Type. If you choose Banner Text, enter the required banner text to be displayed. If you choose File Name, specify the path of the file from which the banner text has to be picked up.
10 Enter the virtual IP addresses as required.    
11 Set appropriate status of WebAuth Intercept HTTPSCaptive Bypass Portal, and Watch List Enable.
12 In the Watch List Expiry Timeout field, enter the time in seconds after which the watch list should time out.
13 Set appropriate status for Disable Success WindowDisable Logout Window, and Login Auth Bypass for FQDN.
14 Check the Sleeping Client Status checkbox to enable authentication of sleeping clients and then specify the Sleeping Client Timeout in minutes. The valid range is between 10 minutes and 43200 minutes.
15 Click the Advanced tab.
16 In the Redirect for log-in field, enter the name of the external server to send a login request.
17 In the Redirect On-Success field, enter the name of the external server to redirect after a successful login.
18 In the Redirect On-Failure field, enter the name of the external server to redirect after a login failure.
19 To configure external local web authentication, perform these tasks: Under Redirect to External Server in the Redirect Append for AP MAC Address field, enter the AP MAC address. In the Redirect Append for Client MAC Address field, enter the client MAC address. In the Redirect Append for WLAN SSID field, enter the WLAN SSID. In the Portal IPV4 Address field, enter the IPv4 address of the portal to send redirects. In the Portal IPV6 Address field, enter the IPv6 address of the portal to send redirects, if IPv6 address is used.
20 To configure customized local web authentication, perform these tasks: Under Customized Page, specify the following pages: Login Failed Page Login Page Logout Page Login Successful Page
21 Click Update & Apply.

Here you can choose a certificate that you want to present guests with when they hit the Captive Portal:

Normally, we would want to have a cert signed by a trusted public CA, but since we don’t have one we won’t select anything in the ‘Trustpoint’ field.

Another thing to point out here is that the Virtual IPv4 Address and Trustpoint certificate must be specified in the global Web Auth Parameter Map. Adding new map won’t even have that options.

Web Auth Config 2

For simplicity, we have just used ‘global‘ Web Auth Parameter Map.

3. Configuring the Web Authentication WLANs

Follow the procedure given below to configure WLAN using web auth security and map the authentication list and parameter map:

Procedure

  Command or Action Purpose
1 enable Example:
Device> enable
Enables privileged EXEC mode. Enter your password if prompted.
2 configure terminal Example:
Device# configure terminal
Enters global configuration mode.
3 wlan profile-name wlan-id ssid-name Example:
Device(config)# wlan mywlan 34 mywlan-ssid
Specifies the WLAN name and ID. profile-name is the WLAN name which can contain 32 alphanumeric characters. wlan-id is the wireless LAN identifier. The valid range is from 1 to 512. ssid-name is the SSID which can contain 32 alphanumeric characters.
4 no security wpa Example:
Device(config-wlan)# no security wpa
Disables the WPA security.
5 security web-auth {authentication-list authentication-list-name | parameter-map parameter-map-name} Example:
Device(config-wlan)# security web-auth authentication-list webauthlistlocal Device(config-wlan)# security web-auth parameter-map sample
Enables web authentication for WLAN.Here, authentication-list authentication-list-name : Sets the authentication list for IEEE 802.1x. parameter-map parameter-map-name : Configures the parameter map. Note  When security web-auth is enabled, you get to map the default authentication-list and global parameter-map . This is applicable for authentication-list and parameter-map that are not explicitly mentioned.
6 end Example:
Device(config)# end
Returns to privileged EXEC mode.
WLAN 1
WLAN Config 1

Layer 2 security we will select none here:

WLAN 2
WLAN Config 2

Foreign WLC policy to Anchor the SSID:

Foreign WLC SSID Policy

Anchor WLC policy for the SSID:

Anchor WLC SSID Policy

4. Configuring Pre-Auth Web Authentication ACL (GUI)

Before you begin

Ensure that you have configured an access control list (ACL) and a WLAN.

Procedure

Step 1 Choose Configuration Tags & Profiles WLANs.
Step 2 Click the name of the WLAN.
Step 3 In the Edit WLAN window, click the Security tab and then click the Layer3 tab.
Step 4 Click Show Advanced Settings.
Step 5 In the Preauthenticaion ACL section, choose the appropriate ACL to be mapped to the WLAN.
Step 6 Click Update & Apply to Device.
WLAN 3
L3 WLAN

We use the pre-auth ACL here that only allows guests access to DNS and DHCP while blocking access to the network until they have authenticated.

Pre-Auth ACL
Pre-Auth ACL

5. Configuring a Local Banner in Web Authentication Page (GUI)

Procedure

1 Choose Configuration > Security > Web Auth.
2In the Webauth Parameter Map tab, click the parameter map name. The Edit WebAuth Parameter window is displayed.
3In the General tab and choose the required Banner Type: If you choose Banner Text, enter the required banner text to be displayed. If you choose File Name, specify the path of the file from which the banner text has to be picked up.
4 Click Update & Apply.

6. Create Local WLC Guest Users Credentials

What was not covered in the Cisco guide was how to add a guest user. To create a new guest user:

Navigate to Configuration > Security > Guest User:

Guest User

Now we have everything configured and a guest user account set up we are ready to connect to the guest LWA WLAN – Woohoo!

iPhone Captive portal
iPhone Connected

Successfully connected client via LWA and anchored to the anchor WLC from the foreign WLC:

Foreign WLC Client

Successfully connected client via LWA and anchored on anchor WLC:

Anchor WLC Client

I was just using the default captive portal in my lab – if you or your customer would like to customise the captive portal here is the Cisco guide and a screenshot of where you can download the webauth bundle from Cisco for your C9800! The bundle is very dated and has a proper vintage feel to it but it’s certainly possible to adjust it so it looks OK 🙂

https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wlan-security/115951-web-auth-wlc-guide-00.html#anc8
Web Auth Bundle

Hopefully, this post helps and saves you guys a bit of time if you need to configure a Local Web Authentication (LWA) WLAN in the future.

As always any feedback or comments are always welcome.

X

WN Blog 013 – Cisco Catalyst 9800-CL – Redundancy HA SSO (CLI and Deeper Dive)

Hey! And welcome to another C9800 blog 🙂 Recently, Matt has blogged about Redundancy configuration using GUI and covered HA/SSO operation here. Today, I wanted to do a deeper dive into the topic. Enjoy!

Lab Environment:

In my lab, both C9800-CL VMs will sit on a single ESXi box. In the real world, they would rather be distributed across separate VM hosts or even separate geographical locations. In any case, configuration would stay identical and same pre-reqs are still relevant. Most importantly, both vWLCs need access to the LAN on their mgmt. interface (that will also be used by APs to register to) and a separate ‘P2P’ L2 link to form & monitor HA and sync configuration.

See a full picture of my lab below:

Lab – Fuill Picture

And to make things simple, also see a simplified logical diagram of C9800-CL networking and interfaces mapping to the VMware vSwitches:

Lab – Simplified Diagram

HA/SSO nitty-gritty stuff

  • HA/SSO in C9800 works differently to how it works in Aironet, it’s more like a VSS now.
  • Initially, both C9800-CL WLCs will require management IP address.
    • Note, that management IP is optional for a secondary WLC if you have a console access to it. Mgmt IP is no longer required to form a HA Pair
  • Redundancy Management as we know it from AireOS (additional IP per WLC from the same subnet as mgmt.) doesn’t exist anymore and therefore Redundancy Port IP is no longer automatically derived from the Redundancy Management IP (it used to be 169.254. + last two octets of the Redundancy Management IP) – you have to manually specify “Redundancy Port” IP and elect one of the VM interfaces to be used as a Redundancy Port (RP)
  • “Redundancy Port IP” is now called “Redundancy Local IP” (GUI) or “Local HA Interface IP” (CLI) <- same thing, just called differently in different places
    • Local HA Interface IP (I’ll call it just that as we’ll use CLI to form HA/SSO in this example) should be in a different subnet to a management interface. This subnet should not be routable.
  • Both WLCs are using the same mgmt. IP to maintain AP CAPWAP during failover once HA/SSO is formed
  • Two units are using a dedicated Redundancy Port (now called HA-Interface) to sync
  • Role election happens at boot, uses priority (1-15, default is 1; higher priority is preferred) or lowest MAC if priority is the same; two controllers are supported, not more
    • It’s best practice to assign higher priority to the preferred active WLC
  • Standby WLC continuously monitors Active via RP keepalives
  • Gratuitous ARP is sent by a Secondary WLC that is taking over the Active role
  • No preempt functionality is supported – failback, if needed, is manual
  • Timers:
    • SSO Failure Detection: 50 ms
    • Reconciliation Time (Standby becoming Active): max 1020ms
  • Reloading Active WLC via CLI: # reload will also reload a standby immediately
  • Forming HA wipes out the config!!!
    • Always form a HA first, configure later
  • WLC Interface used for HA will disappear from the interfaces list (sh ip int br etc. will not list it anymore)

HA/SSO Pre-reqs

  • HA Pair can only be formed between two WLCs of the same form factor (C9800-CL VMs in this example)
  • Both WLCs must be running same code version
  • Max RP RTT = 80ms, min bandwidth = 60Mbps, minimum MTU = 1500
  • Both WLCs are built, accessible, VMware networking is configured, WLCs have access to the LAN and separate L2 link for the HA is available between the WLCs
    • C9800-CL HA-Interface that we intend to use as a Redundancy Port L2 HA inter-vWLC link must be put into a separate, unused VLAN! Using existing VLAN/subnet, especially if it’s management VLAN/subnet, would theoretically work (I’ve tested it) but it will cause instability, duplicate ARP entries and ARP resolution issues. See below the ‘show logging’ output from my switch, where I had HA-Interfaces IP sitting in the mgmt. subnet:
  • See full deployment guide of the C9800-CL for VMware ESXi in our previous blog here
  • Prepare IP addresses and subnets; in this example we will use the following:
    • Active WLC
      • Chassis 1, priority 2
      • Mgmt. (VLAN 11): 10.10.11.35 /24
      • HA Local IP (VLAN 666): 10.10.66.35 /24 (VLAN 666
    • Backup WLC
      • Chassis 2, priority 1
      • Initial Mgmt (VLAN 11): 10.10.11.40 /24 (optional if have console access, won’t be used anymore once HA is formed)
      • Mgmt: (VLAN 11) 10.10.11.35 /24 (shared between HA WLCs once HA is formed)
      • HA Local IP (VLAN 666): 10.10.66.40 /24

HA/SSO Configuration

I feel CLI gives us better visibility into what is happening on both WLCs while forming HA Redundancy, therefore we will use CLI here. It is possible to use GUI too, but I am not a huge fan of “click, wait, pray and hope for the best” approach, where I can’t see what’s happening 🙂

CLI

  • Active C9800-CL
    • # chassis ha-interface Gig 3 local-ip 10.10.66.35 /24 remote-ip  10.10.66.40
    • # chassis 1 priority 2
    • # wr
    • # reload
  • Standby C9800-CL
    • # chassis ha-interface Gig 3 local-ip 10.10.66.40 /24 remote-ip  10.10.66.35
    • # chassis 1 renumber 2
    • # chassis 1 priority 1
    • # wr
    • # reload

GUI

Refer to our previous C9800-CL Redundancy blog written by Matt here

Note: Standby WLC would need to be accessible on its management IP address to use GUI for Redundancy configuration.

Tshoot & Validation

It is good and useful to know how to monitor, tshoot and validate HA Redundancy configuration. You can do it in both CLI (see below) and GUI (Monitoring > System > Redundancy; Dashboard > Slot: Active / Standby)

Chassis

# show chassis

Show Chassis
  • Validate Chassis# and Priority. Also see the HA state (“Ready” is the final and desired stage, you can also see “Initializing” etc.) and HA-Local IP of both C9800-CL VMs.

# show chassis ha-status local

Show Chassis HA-Status Local
  • Check before the reboot while forming the HA
  • Active should have higher priority, HA Local and Remote IPs and assigned interfaces are shown, together with Chassis# and Priority that will be assigned to this local WLC upon reboot (you might have changed those values and it can be slightly confusing, so it’s best to check after configuring and before rebooting_

Redundancy

# show redundancy

Show Redundancy
  • Another useful show command, where we can validate uptime in current state, versions, modes and more!

Want to Know More?

Yeh, totally get it – me too 😉 I’ll drop some useful stuff that I came across and used at some point here that might help with configuration and troubleshooting.

Manual failback

  • # redundancy force-switchover
  • Can be very useful in scenarios like primary DC maintenance, where Backup C9800-CL takes over. Since there is no preempt mechanism in C9800 HA (former Active WLC won’t claim its original Active role back when it comes back online)

Clear HA Configuration

  • # chassis clear
  • # reload

This is how it looks in the CLI:

Note: it’s enough to clear chassis information on the active HA Pair WLC, despite the information that the ‘other’ (Backup) chassis will keep its HA configuration. I’ve tested it and it will not! Backup chassis will also reboot with clear redundancy config!

  • Chassis clear is used to break the HA; be careful with this as you will end up with duplicate IPs! Both former Active and Passive WLCs will share identical configuration upon reboot!

Duplicate IP addresses after clearing chassis config

Successful HA/SSO Formation

This is how successful HA  formation should look like in a console of both Active and Passive WLCs:

Successful HA/SSO Formation upon reboot once ha-interface was configured; Active (left) was elected Active due to higher priority

That’s it for now! It feels great to play around with new toys and discovering differences between old and new WLCs! Don’t hesitate to comment, add to the discussion or simply let the world know that you are enjoying beta-testing Cisco gen 1 products as much as we are 😉

WN Blog 011 – Cisco Catalyst 9800-CL – Redundancy HA SSO (GUI and Basics)

Hey!

Welcome to another one of our blogs on configuring the new Cisco Catalyst 9800 WLC.

This time we are going to take you through configuring 2 x C9800-CLs for redundancy HA SSO. 

First here is an overview of my home lab setup:

Matts Lab

I currently have 2 x ESXi servers and a C9800CL on each of them – what it is important to point out below here is that I have VLAN 12 configured to use for my L2 redundancy ports between the WLCs.

ESXI Servers vSwitch Config

Interface Gigabit Ethernet 3 will be used for the L2 HA in this setup:

ESXI C9800 Network adapters

Just want to point out here that at this stage we have 3 x interfaces – Gigabit Ethernet 1 – 3:

C9800s Ethernet Interfaces

I then began the redundancy configuration on both of the WLCs.

On the primary WLC I specified the “local IP” as the IP address I had just set up on VLAN 12 and the remote IP address of the secondary WLC that I had just created on VLAN 12.

HA interface I have used Gigabit Ethernet 3.

I wanted the WLC on the left to be the primary WLC so I set the active chassis priority to higher than the secondary WLC on the right:

C9800s Redundancy Config

After I applied the configuration I then saved the config and reloaded both of the WLCs at the same time, crossed my fingers and prayed to the wireless networking gods! 😀

C9800s Save & Reload

A few minutes later…

C9800s Successfully in Redundancy HA SSO 1

We can see now that the WLCs have rebooted and successfully formed an HA SSO pair. You can now also see a new dropdown on the dashboard to flip between active and standby stats:

C9800s Successfully in Redundancy HA SSO 2

Standby stats:

C9800s Successfully in Redundancy HA SSO 3

Note the G3 interface is gone after forming a HA:

C9800 No Gigabit Ethernet 3
C9800 No Gigabit Ethernet 3 GUI

Also note that HA/SSO is required to take advantage of a very nice new featur of the C9800 series WLCs, which is the “always on” feature from its hitless upgrades.

Here is how it works:

  • The controller automatically selects groups of APs that can be upgraded, while other nearby APs will still provide coverage to the clients
    • RRM is used to determine AP neighbors that can provide redundant client coverage
    • The aggressiveness of these groupings is configurable.
      • You can have many groups (few APs per group), with very minimal coverage impact, but it will take a long time to complete.
      • Or you can have fewer groups (more APs per group) with a greater chance for coverage impact but will complete much more quickly
  • The secondary Controller is upgraded to the new software version and rebooted
  • The controller uses 802.11v to shuffle clients away from the APs in the first group so that they can be rebooted without impacting the clients
    • Clients not supporting 802.11v will get ungracefully kicked off the AP
  • The controller moves those APs to the new controller, thus upgrading the AP code when they join
    • Once upgraded and controller-joined, clients may join these APs
  • The same process is automatically repeated for all successive groups of APs
  • Once all APs are moved to the N+1 controller, the code is upgraded on the primary controller and it is rebooted
  • Once the primary controller is back online, the APs can optionally be moved back to the primary controller

There you go – that is how you set up and configure your virtual C9800CLs for HA/SSO – hopefully this blog saves you a bit of time if you ever need to do something similar!

PS. Shout out to Ashley Georgeson who helped with this 🙂

WN Blog 010 – Cisco Catalyst 9800 – Configuration Guide (Basics & Central Switching)

Finding C9800 stuff hot and interesting? You’re not alone out there! 🙂

After initial C9800 configuration (see this blog) and registering your first AP (our AireOS AP Join Blog is still applicable here) you’re pretty much ready to go! There are just some pre-reqs to consider before we can associate with our first C9800 BSSID.

Pre-reqs

  • C9800 is built, GUI and CLI access is working
  • AP is registered
  • AP mgmt. VLAN (VLAN 10 in this example) is operational, AP has connectivity with to at least the vWLC
  • VM mgmt. VLAN (VLAN 11 in this example) is operational, C9800 can communicate with the network
  • WiFi Users VLAN (VLAN 20 in this example) is operational and, since it’s central switching we talk about here, VLAN 20 is allowed on C9800-CL management trunk (Gi2 in this example), on Port Group on the vSwitch C9800 Gi2 is mapped to and on ESXi management trunk to the switch; VLAN 20 must be allowed across entire path from the C9800 to its gateway

Note: since we’re talking central switching mode here, authenticated WiFi users’ traffic will be dropped into WiFi users VLAN at the vWLC level!

Lab Environment

To get better picture, see my lab environment below:

Lab Environment

New C9800 Architecture

Even though C9800 offers pretty much features parity with our beloved AireOS WLC, it is slightly different. If you spent some quality time with Cisco DNA Center you will feel comfortable with the new GUI and structure. If not (like me), be prepared to change some old habits and approach C9800 with an open mind 🙂

C9800 is designed to fit perfectly into Cisco SDA world and integration with DNAC and use of SGTs. But don’t worry if you don’t have few DNAC boxes worth £70k each and a fabric underlay lying around. C9800 can still work in a traditional way. The only real difference is where we configure the same stuff as we did in AireOS (using Profiles) and how we stich it together (using Tags).

Look at the visual representation of major blocks building a basic WiFi setup.

Wireless Setup Flow Overview

You can create multiple WLAN Profiles (SSIDs) and Policy Profiles and correlate both using a single Policy Tag. You then choose your AP Join Profile and Flex Profile (if in use) and correlate both using Site Tag. Your 2.4 and 5GHz RF profiles are stitched together using RF Tag. You now have all the config needed for the AP and SSID to operate described with just 3 tags, that are finally assigned to the AP of your choice. Takes some time getting used to, but it’s really not too bad.

Let’s dive straight in. Once we start looking at the real config it shouldn’t feel too overwhelming.

Note: only the WLAN Policy is mandatory but it is best practice to use all 3 policies and understand impact they all have on our wireless network.

Configuration

In this example we’ll configure simple WPA2-PSK WLAN and cover more interesting stuff like popular enterprise security approaches using EAP (PEAP, EAP-TLS, BYOD) in future blogs! Guest flow (CWA with ISE) already has its blog and you can find it here: https://wifininjas.net/index.php/2019/08/13/wn-blog-009-c9800-wlc-guest-mab-cwa-ise/ 🙂

Below steps show how to configure your first centrally switched WLAN. It’s PSK-based, but you can easily translate it to any other auth method. Have fun 🙂

1. Configure WLAN Profile

This is a pretty standard WLAN configuration that we’re all familiar with. Quite a few boxes to tick, tons of possibilities, standard Cisco overcomplication that feels like 1:1 port from the AireOS. Simplifying opportunity missed. But let’s not digress!

Go to Configuration > Tags & Profiles > WLANs or (I like to do that), Advanced Wireless Setup (top-right corner), where you can switch between all main solution building blocks from one list.

Advanced Wireless Setup

Amazing, isn’t it?

In General tab, configure Profile Name (must be unique) and SSID name (can have more than one on the same WLC) along with WLAN ID (ID <=16 = included in default Policy Profile, ID > 16 = won’t be included in default Policy Profile), Radio Policy (2.4GHz, 5GHz, or both). As L2 security in Security tab, use WPA + WPA2 with AES encryption and PSK key mgmt and specify ASCII PSK Key. FT should be disabled here due to Krack vulnerability. Leave all other security details on their default values. Advanced tab settings can also be left on default but please be mindful of the impact some settings on the performance and compatibility of your SSID. This is a topic for another discussion, but in essence I’d say that:

  • Aironet IE enables SSID name inclusion in beacons among many other things, which is helpful AF but can affect clients compatibility with this SSID
  • P2P Blocking is normally enabled on Guest SSIDs (blocks WiFi clients from attacking each other)
  • 802.11v can seriously affect (positively or negatively) your roaming behaviour – test with your clients
  • 802.11k is generally a good feature to use as it should reduce time your clients spend scanning off channel for alternative APs while roaming
  • Band select can be useful in some scenarios, but steer away when used with voice as it will make roaming slower
WLAN Profile General tab
WLAN Profile Security tab

2. Create a VLAN

Since we’re dealing with central switching, we might think that we would need a dedicated SVI on the C9800 sitting in the VLAN into which we would like to drop authenticated users’ traffic in the same fasion as we did in AireOS. We could do it and it would work, but it’s not necessary here. All we need is to create VLANs (as opposed to SVIs) to get centrally switched WLANs to work.

Add VLAN in CLI ((config)#vlan [VLAN ID]; (config-vlan)#name [VLAN name]) or GUI in Configuration > Layer2 > VLAN > Add

In this example I’m using VLAN 20 for wireless users. L3 sits on my core switch. VLAN 20 is allowed on vSwitch and ESXi trunk. Refer to the lab network diagram for more clarity – one pic speaks thousand words.

Add VLAN for wireless users

3. Configure Policy Profile

This is where we can configure TrustSec, decide if we want to use Central Switching, Authentication, DHCP or Association, map WLAN to a VLAN, apply an ACL to an SSID, turn on RADIUS Profiling, specify QoS, AVC, CAC, Anchors, AAA Policy (attributes returned to the NAC), basic WLAN timers and many more. Sounds easy? I know, I too feel it’s slightly on the overcomplicated side 🙂

General Tab:

Policy Profile – General Tab

Minimum config required:

  • Specify Name
  • Set Status to ENABLED
  • Enable Central Switching and Central DHCP

If you want to know more:

  • WLAN Switching Policy can move certain roles (switching, auth, DHCP, association) between AP and WLC. Central everything is the default. Note that “Central DHCP” must be used in Central Switching mode. If you disable it, clients won’t get an IP at all. Authentication and Association roles can be moved between WLC and AP and either would work just fine.
  • Passive Client (in short) allow comms from wired devices to wireless devices with static IP address by enabling WLC to pass ARP requests to them. Normally WLC proxies that responses but since there is no DHCP used, WLC wouldn’t know about those devices.

Access Policies Tab:

Policy Profile – Access Policies Tab

Minimum config required:

  • Specify VLAN/VLAN Group assignment (use just a VLAN number or use VLAN name set in step 2.)

If you want to know more:

  • VLAN/VLAN Group is where we specify the VLAN we want to drop wireless users into! This is extremely important to understand, since we don’t assign WLANs to dynamic interfaces (central) nor we have VLAN to WLAN mapping (flex) anymore. I have assigned ‘LAB-WIRELESS-USERS’ VLAN 20, that I created in step 2.
  • WLAN ACL can be applied to do basic firewalling on the AP or WLC level (depending if using central switching or Flex) before the traffic even hits the firewall.

Let’s leave the rest on default for now. We’ll cover it in more details in future blogs.

4. Configure Policy Tag

Policy tag just stitches WLAN Profile and Policy Profile together, nothing else 🙂

We could mix and match different WLAN and Policy Profiles while creating different tags. Example:

  • PSK WLAN with Policy Profile on VLAN 20
  • EAP WLAN with Policy Profile on VLAN 20
  • MAB WLAN with Policy Profile on VLAN 666

You get the drill.

Policy Tag

Minimum config required:

  • Create a new Policy Tag and select WLAN and Policy Profiles created in previous steps

5. Configure AP Join Profile

Here we’ll configure basic parametres used by APs after they join a controller.

AP Join Profile – General tab

Minimum config required:

  • Specify AP Join Profile Name

If you want to know more (I’ll only mention stuff I think is useful and filter out the noise):

  • If setting up a quick SSID is all we need, we can just create AP Join Profile, and specify its name in General tab. All other settings within the AP Join Profile can be left on default.
  • CAPWAP tab allows us to adjust CAPWAP & Retransmit timeouts and hardcode Primary and Secondary Controller Name and IP for N+1 designs.
  • AP tab allows us to set a Country Code, AP EAP Auth (FAST, PEAP, TLS), enable Hyperlocation, adjust BLE Beacons and AP Packet Capture settings (can capture straight to FTP location which is quite cool).
  • Management tab allows us to specify specific AP code that AP should download upon Joining the WLC, enable AP SSH access, set local mgmt user and dot1x credentials and set thresholds for detecting rogue APs.

Note: Flex Profile configuration is not required in Central Switching architecture.

6. Configure Site Tag

Now, Site Tag is slightly special 🙂

You might expect that since Policy Tag was just stitching profiles together, why would Site Tag be any different?

In Central Switching mode we don’t even specify Flex Profile assignment (let’s discuss it in more details in C9800 Flex blog post). We just specify Site Tag Name, AP Join Profile and, most importantly, we specify if the site is Local (centrally switched) or not (flex). It is confusing AF! In our example, we configure Site Tag with “Enable Local Site” checked.

Site Tag configuration

Minimum config required:

  • Set Site Tag Name, specify AP Join Profile configured in previous step and check “Enable Local Site” (which means that AP will leverage central switching).

7. Configure RF Profile

Custom RF Profiles are optional but extremely helpful to have, especially if your C9800 WLC caters for sites that have different RF requirement coverage areas. Maybe some sites will have APs expected to work in a High Density or Voice environment, while most APs would be serving ‘just data’ hungry stations? We would create different RF profile for each coverage type (High Density, Voice, Data) and for each band (2.4 and 5GHz).

Minimum config required:

  • None – custom RF Profiles are optional

If you want to know more:

  • General tab is just for specifying Name, Band and Description. Make sure your RF Profile is Enabled.
  • 802.11 tab is used to specify Data Rates and MCS Rates. Disabling low data rates is crucial in most scenarios (I personally never had to allow low data rates but I’ve heard stories where others had to due to weird wireless stations requirements). In short, management and control traffic is exchanged at lowest mandatory data rate. Clients associated with our BSSID would rate shift down when moving away from the AP and when the SNR drops for any reason. Supporting low rates and setting mandatory rate too low would contribute to sticky clients behaviour, slower transmission speeds for dodgy clients sitting on low data rates, increased airtime utililisation by those clients and finally, low mandatory rates would contribute to more airtime being consumed for control and management frames, leaving less space for data transmissions. Generally, in most situations, I would set minimum mandatory rate to 12 (standard deployment) or 24 Mbps (higher density), disable all lower rates and set all higher rates to supported. Please note that proper RF design following a proper on-site survey and predictive survey (and sometimes pre-deployment survey with AP-on-a-stick) is extremely important when tweaking radio configuration! Crap in crap out, can’t mitigate shit design with good config.
  • RRM tab is quite a comprehensive topic on its own! Let’s try to keep it short. You might want to increase default number of clients per AP that generate traps. Transmit Power Control (TPC) is extremely important to use wisely and to our advantage. Normally, it’s best to try and match Tx power level of the least powerful wireless device in the network. Allowing maximium available Tx power on the AP won’t necessarily mean ‘better coverage’. Coverage works in both directions – from AP and from a client. Not matching Tx power between AP and a client might cause asymmetric traffic patterns, where client can hear the AP, but AP can no longer hear the client. You also don’t want to go too low with Tx power as the goal is to maintain as high MCS rate as possible. Normally, 2.4GHz AP max Tx should be considerably lower than 5GHz max Tx. Dynamic Channel Assignment (DCA) is another supremely important factor contributing to general performance of a WiFi network. Do we want to bond channels? Do we want to use all available 5GHz channels there? The answer is “it depends”. I’d consider using 40MHz only in smaller deployments and only if it wouldn’t contribute to excessive channel contention. Remember that doubling the channel width doubles the interference and reduces SNR by 3dB. Also, even without ‘our’ APs, RF spectrum might already be crowded so we might want to stick to 20MHz sometimes even in smaller sites. Generally speaking it seems wise to stick to 20MHz in most cases in the enterprise world (but not only!) unless sheer higher throughout is required and you fully understand implications of bonding channels. Also, newer use “Best” channel width as we’ve seen it going wild with using 80MHz in central London on sites with 20+ CCI in a busy multi-tenant building. Now, which UNII bands to use? Stick to non-DFS channels 36-48? Use 36-64? Or maybe use all 19 available (in the EU) channels? It’s hard to answer this with a ‘one size fits all’ type of an answer, but in most cases we’d use all UNII bands, unless we have some weird clients that don’t work too well with upper bands and/or don’t support 802.11k while quick roaming is important.
  • Lastly, there are features like RX SOP or Client Load Balancing in the Advanced tab. RX SOP solves half of the CCI issue by ignoring signal below configured thershold from neighbouring BSSIDs operating on the same channel on an AP level. But how about contention on the wireless station level? Exactly. It’s still there. To solve this issue better, we’d want to think about 802.11ax, where OFDMA and BSS colouring should combat that contention crap nicely. Final tip – be careful with Client Load Balancing as it might hard disassociate your precious clients, potentially making them unhappy, especially if they’re in the middle of a call or conference. I now feel we should have a separate blog and podcast about the RF Profiles and RRM. And we probably will!

8. Configure RF Tag

We are almost there! With RF Profiles sorted, we just need to stich 2.4 and 5GHz profiles together with an RF Tag.

RF Tag configuration

Using RF Profiles and RF Tags, again, is optional (but cool, so please use it).

9. Apply Tags to the APs

Yup, this is the last step.

Select APs to tag and tag them with three tags we have created above (out of which only Policy Tag is mandatory).

Apply tags to APs

Think of tagging APs as of adding them to their relevant AP Groups with specific WLANs and RF Profiles. Well, kind of 🙂

That’s it. Simple. Similar to the AireOS that we all know and love so much, isn’t it? 🙂

Thanks for surviving going through this lengthy post and see you in the next WiFi Ninjas Blog!

WN Blog 009 – Cisco Catalyst 9800 – Guest MAB CWA ISE Config

Hey!

Welcome to another one of our blogs on the configuration of the new series of WLC from Cisco the C9800!

In this post, I want to go through with you an issue that I ran into when configuring a Guest SSID which was using MAB with a CWA to redirect to a portal on ISE. 

A high-level overview of the C9800 -40 + 3800i APs – Local mode, Central Switching & Authentication. ISE was configured correctly and was working correctly as it should of the AireOS 5508 that I was replacing and was still working.

I had followed all the steps & configured everything in this Cisco guide apart from the BYOD flow as that was not a requirement for this project.

Cisco guide: https://community.cisco.com/t5/security-documents/ise-and-catalyst-9800-series-integration-guide/ta-p/3753060

But I was hitting two scenario issues – the first one was that I was not being redirected to the portal when connecting to the SSID but I was authenticated on ISE and had internet. The second was that I was being redirected and could authenticate with ISE by inputting the code but then not getting any internet after.

The configuration for the first scenario where I was not getting redirected but I was authenticated and had internet was that I had created the “Redirect_Webauth_ACL” and that was applied globally on the WLC – very much the same as you would on AireOS.

The configuration for the second scenario where I was being redirected but then not getting any internet was that I had applied the “Redirect_Webauth_ACL” to the “WLAN ACL” in the “Access Policy” of the Guest “Policy Profile”

So even though I had followed the documentation neither scenario was working how I would expect it to. I am going to take you through below in some screenshots the config I had applied and where as well as show you how I managed to get it working even though what I did was not clear from the Cisco guide.

One thing to call out here for when you come to write an ACL on the C9800 is to remember that they use the IOS syntax instead of what you would be used to on the AireOS WLCs.

Cisco 9800 Guide Notes
Cisco 9800 Guide Notes

From the Cisco guide this is an example of how to write the web auth redirect ACL – Cisco ACL example for the C9800:

Cisco Guide ACL
Cisco Guide ACL Example

This is where you configure the ACLs and can see that the ACL that I had configured for the web auth redirect is called “GUEST_REDIRECT_ACL”

9800 ACL Overview of all ACLs highlighting the Guest
9800 ACL Overview of all ACLs highlighting the Guest Redirect

We can have a look at the redirect ACL rule here and can see that I have specified the two ISE servers and DNS (I had previously made the ACL more specific but after many hours of troubleshooting I decided to make it bit more open)

Guest Redirect ACL
Guest Redirect ACL

Now I want to show the WLAN config where you can see the Authorisation & Authentication lists that have been specified are the two ISE servers:

Guest WLAN Security 1
Guest WLAN Security 1
Guest WLAN Security 2
Guest WLAN Security 2

Now in this scenario where I was being authenticated but not redirected, in the policy profile for the guest I had not specified the redirect ACL here.

Guest Policy Profile without ACL
Guest Policy Profile without ACL applied

When I did specify the redirect ACL in the access policy above I was now being redirected but then was not getting any internet.

Checked the guide again to make sure everything was correct which it seemed it was so left me scratching my head at this point as to why was not working as expected.

So I reached out to my security friend Aref who skills far surpass mine when it comes to security & ISE to double-check ACL config & ISE policies for the Guest Wireless MAB.

Here are a few screenshots of how ISE is configured:

ISE Config 1
ISE Config 1
ISE Config 2
ISE Config 2
ISE Config 3
ISE Config 3
ISE Config 4
ISE Config 4
ISE Config 5
ISE Config 5

So Aref confirmed that all looked good from an ISE configuration perspective – so how did we get it working I hear you ask, great question! What we had to do was to specify another ACL which we called “DENY_GUEST_INTERNAL” which in this rule we basically blocked any access to RFC 1918 but then allowed anything else and we applied this ACL to the Guest “access policy” in the “policy profile” with the redirect just applied at a global level and now we finally got redirected as well as internet!

Here are some screenshots of the other ACL, its configuration and where we applied it:

C9800 ACLs overview highlighting Deny Guest Internal
C9800 ACLs overview highlighting Deny Guest Internal
Deny Guest Internal ACL
Deny Guest Internal ACL
Policy Profile with ACL applied
Policy Profile with ACL applied

It was quite a long day of troubleshooting and trying different scenarios before we managed to finally get it working as expected and I feel that the way we did finally manage to get it working was not clear from the Cisco documentation so hopefully this can help save you guys some time if you have to configure a guest network with CWA + MAB and run into the same scenarios as I did.

Hope you enjoyed this blog on another configuration gotcha from the C9800 – as we deploy more of these and find anything else that we think may help others who will be implementing these for the first time we will post more blogs with our findings!

🙂

WN Blog 008 – How to Change Cisco CMX GUI “admin” Password

Does this look familiar?

GUI password no longer works 😉

Sometimes we forget a password, sometimes it’s set by someone else and they forgot it, left company or are just evil and won’t tell us 😉

Assuming you still have access to  SSH, VMware console (if virtual), Remote or physical serial console (if appliance), you can easily change GUI password of your CMX box.

Console access: SSH to the left; VMware console to the right

NOTE: CMX uses separate user accounts for GUI and CLI access. GUI uses “admin”, CLI uses “cmxadmin” for all administrative purposes.

Since we want to regain access to GUI, we’ll be resetting “admin” user password.

First, we can list users for added peace of mind. Then, we can choose our admin user and set a new password.

GUI “admin” user password change syntax:

  • [cmx ~]$ cmxctl users list
  • [cmx ~]$ cmxctl users passwd admin
  • Enter a new password for user admin: [new password]
  • Repeat for confirmation: [new password]

And a quick screenshot:

GUI “admin” user password change

If you require more details, please see this link: https://www.cisco.com/c/en/us/td/docs/wireless/mse/10-5/cmx_config/b_cg_cmx105/performing_administrative_tasks.html

If you typed incorrect password too many times, you’ll need to unlock CLI or GUI user.

To unlock CMX access for a CLI or GUI user after they have been locked out, use the cmxctl users unlock command.

[cmx ~]$ cmxctl users unlock { cli username | gui username }

That’s it! Access regained. Beer time 😉