Introduction to SPAN with Cisco ACI
With ACI, there are 3 types of SPAN, which defines what type of traffic can be monitored and depends on the type of port you want to include.
- Fabric: Use this type of session when you want to monitor traffic that's being sent from leaf to spine.
- Access: Use this type of session when you want to monitor traffic that's being sent from endpoints to the leaf.
- Tenant: This type of session will monitor traffic for a particular EPG, regardless of the physical ports where that EPG is effectively present.
There are some considerations to take into account for every scenario. I'm going to set out according guidelines as well as describe configuration steps in the following sections.
Access SPAN can accept both local SPAN and ERSPAN as destination. Local SPAN simply means that destination will be a physical port in the fabric. ERSPAN defines a new encapsulation (GRE) for monitored traffic. There are 3 different versions of ERSPAN: type I, II and III. If you have the latest Nexus 9000 hardware (EX), then type II is used for all types of SPAN. If the hardware is based on Broadcom Trident 2 (T2) ASIC, then access and tenant SPAN will leverage ERSPAN type I. In addition, the destination IP used for ERSPAN must be learned by the fabric. In other words, the SPAN destination has to be configured under an EPG and unicast routing has to be enabled so the fabric knows both IP and MAC of the endpoint.
There are another couple of guidelines and limitations:
- If interfaces defined as sources are located under multiple leaf switches, then destination group can only be ERSPAN.
- Likewise, If source is a vPC interface, then destination group can only be ERSPAN.
- When local SPAN destination is required, both source and destination group must be on the same leaf node.
Now, let's create a SPAN session for the following topology:
As we're going to create a session with multiple ports on multiple leafs as sources, we must use ERSPAN for the SPAN destination. This means the destination type will be EPG, since the station running Wireshark must be learned by the fabric. So first you have to create an EPG for this station and check that the endpoint IP and MAC are effectively learned.
The fabric just needs to forward packets to it, so there isn't a requirement for the source to be able to reach the destination station. Only the fabric must be able to reach it, therefore we must configure an IP on the BD that is linked to the station EPG so the fabric can act as its default gateway.
- Check destination EPG has been learned by the fabric.
- Create SPAN Destination Group under Fabric>Access Policies>Troubleshooting Policies>SPAN>SPAN Destination Groups.
- Create SPAN Destination, within the same window. Select the Tenant/ANP/EPG containing the SPAN destination and specify its IP address. A source prefix is required for every node to use a different source IP in the outer ERSPAN IP header. If single IP is specified, then all nodes will use the same IP.
- Create SPAN Source Group under Fabric>Access Policies>Troubleshooting Policies>SPAN>SPAN Source Groups and select the destination group that we've just created. Make sure the Admin State is set to Enabled.
- Create SPAN Source, within the same window. Add 2 different source paths, and apply a filter defining the EPG (or VDS port-group) you want to monitor. As depicted on the diagram above, leaf-101:eth1/8 needs to be filtered with EPG "app" and leaf-102:eth1/7 needs to be filter with EPG "web".
- Finally select your source group and check that administrative state is enabled and that operational state is up
Now you can launch Wireshark on the monitoring device. Don't forget to enable "ENFORCE to decode fake ERSPAN frame" under Edit>Preference>Protocols>ERSPAN (required for ERSPAN type I). Here is the result when "web" VM (192.168.100.10) wants to connect to "app" VM (192.168.100.11) on port 80:
Tenant SPAN is very useful for monitoring a particular EPG as the fabric completely abstracts the dependency on physical ports. SPAN sessions will monitor any traffic that leaves or is destined to a particular EPG.
The configuration is very straight forward:
- Under a user tenant, navigate to Troubleshoot Policies and repeat the steps of the Access SPAN configuration. The only difference lies in that you'll have to configure an EPG as a SPAN source, rather than physical ports. No filters are available.
Fabric SPAN will monitor VXLAN frames between spine and leaf nodes. It is also leveraging ERSPAN and requires physical fabric ports as source and an endpoint IP as the destination (again it has to be learned by the fabric, thus has to live within an EPG).
To configure fabric SPAN, navigate to Fabric>Fabric Policies>Troubleshoot Policies>SPAN. You'll find the same options as before, but this time you'll be able to create a filter with a particular VRF or Bridge Domain (not both) when configuring the SPAN source. In addition, as source paths, you'll only have the ability to select fabric ports (uplink). ACI derives VNID from bridge domain for L2 traffic and from VRF for L3 traffic. So for routed traffic, select VRF as the filter.
In addition, the extra step with Fabric SPAN is that you have to decode VXLAN frames. And you have to be aware that ACI uses a specific UDP destination port for VXLAN, which is port 48879. (Try to code it in HEX and that will remind you of a very old Cisco practice :-) ).
So in Wireshark, go under Analyze>Decode As. Add a new line with the following settings:
Also, because Fabric SPAN is using ERSPAN type II, you have to uncheck the "ENFORCE to decode fake ERSPAN frame" option if you had it previously enabled.
Example of non-decoded VXLAN frames, UDP 48879 in transport headers:
When frames are decoded you can see the real IP of VMs talking together, and the protocol used:
SPAN with APIC as Destination
When using the Troubleshooting Wizard under the Operations tab, there is also the possibility to run SPAN sessions. As an extra option, and under the condition that you have inband management configured, it's possible to export a pcap file directly to the APIC. This may be very handy if you want to download it locally to your computer to send it to TAC or to add it to some documentation.
Troubleshooting Wizard in action, see the SPAN option:
Set APIC as SPAN destination and download PCAP file: