Date created: Monday, April 25, 2016 10:49:18 AM. Last modified: Friday, June 29, 2018 9:17:18 AM
6500/7600 FIB TCAM Allocation
Contents
FIB TCAM Overview
Show Commands
Examples
Understanding "Show" Outputs
6500/7600s with 3BXL and 3CXL RSPs and 3BXL and 3CXL DFCs support a maximum of 1024K FIB TCAM entries. These 1,024,000 entries are each 72 bits in size and need to be split between IPv4 prefix entries, MPLS label entries, IPv6 prefix entries and mixed IPv4 & 6 multicast entries. The IPv4 entries and MPLS entries are 72 bit entries so each IPv4 route entry (or MPLS entry) uses one FIB entry space. Each IPv6 or Multicast entry uses 144 bits (double the size) so one IPv6 prefix entry uses the space of two IPv4 entries.
Under the default configuration the FIB TCAM is split as 512K entries for IPv4 and MPLS (dynamically shared between those two) and 256k entries for IPv6 and Multicast (also dynamically shared between those two).
Example of SUP720:
Note: The VS-S2T-10G-XL supervisor engine and DFC4XL modules support 1,000,000 routes that are dynamically shared between IPv4 and IPv6 by default.
Note: Adjusting the TCAM requires a reboot in order for the changes to take effect!
Note: In the case of dual-SUPs/RSPs/VSS, when adjusting the FIB TCAM boundary ensure the config register on the backup SUP is set to the same value as the active SUP before rebooting to apply the changes. Otherwise at bootup the backup SUP might not read in the configuration to adjust it's FIB TCAM allocation simply dropping into a passive "slave" mode to the active SUP/RSP which has read the configuration and tried to adjusted its TCAM, although there will be a discrepency between the two RSPs/SUPs so the TCAM won't be adjusted on either. In such a case, check if the backup SUP has a confreg is 0x0 or anything other than 0x2102, "remote command standby-sp show bootvar". To force the confreg values to be the same, try the following trick on the active SUP: "conf t; config-register 0x0; end; copy run start; conf t; config-register 0x2102; end; copy run start".
Note: If the number of TCAM entries reaches 100% (full) then no more routes can be installed into hardware. Packets with no matching hardware route are then software switched. At this point an MLS cef exception flag is set and the flag cannot be cleared until the device is rebooted (even if the TCAM entry count is lowered to below 100% and all packets are being hardware processed again).This is the case for SUP720, RSP720 and SUP2T. The command "mls cef error action [ reset | freeze | recover]" (reset is default) can be used to specify the action taken in the event of a FIB exception (but the flag can still never be cleared without a reboot. Freeze - Allows each protocol to freeze the FIB error. This captures the info and helpful in event of Cisco TAC troubleshooting during system problem. Recover - Allows each protocol to recover from the generic FIB error or a protocol-specific FIB fatal error. Reset - Allows each protocol to clear the data structures and reload the FIB.
Use the mls cef error action freeze command to allow each protocol to freeze the update of the FIB and to debug the condition. Use the mls cef error action recover command to set up the registries to inform the protocol of the error condition and coordinate with the protocols to clear up the data structures and reload the FIB. Use the mls cef error action reset command to clear the data structures and reload the
FIB when a FIB error is detected.
#show mls cef exception status Current IPv4 FIB exception state = FALSE Current IPv6 FIB exception state = TRUE Current MPLS FIB exception state = FALSE
For the SUP2T only, it stores routes as a data structure in a memory pool, which means that it isn't a fixed number of route entries, but is based more on the prefix distribution and how routes are added/deleted. It is possible to run out of space before hitting datasheet numbers (although unlikely). An example is discussed in the following post by a former TAC engineer, a reboot is usually required so that the prefix trie is rebuilt: http://puck.nether.net/pipermail/cisco-nsp/2018-January/105620.html
show run | i maximum-routes show ip cef summary show mls cef summary show mls cef maximum-routes show mls cef exception status
show cef table show platform hardware capacity forwarding
Show the current TCAM allocation, the usage and if there is an exception:
7606-S-RSP720-3CXL-10GE#show mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :-
-------
IPv4 + MPLS - 960k (default)
IPv6 - 16k
IP Multicast - 16k
7606-S-RSP7203CXL#show run | i maximum-routes
mls cef maximum-routes ipv6 16
mls cef maximum-routes ip-multicast 16
7606-S-RSP7203CXL#show mls cef exception status
Current IPv4 FIB exception state = FALSE
Current IPv6 FIB exception state = FALSE
Current MPLS FIB exception state = FALSE
7606-S-RSP7203CXL#show mls cef summary
Total routes: 714403
IPv4 unicast routes: 638075
IPv4 Multicast routes: 4
MPLS routes: 76303
IPv6 unicast routes: 17
IPv6 multicast routes: 3
EoM routes: 4
7606-S-RSP7203CXL#show platform hardware capacity forwarding
L2 Forwarding Resources
MAC Table usage: Module Collisions Total Used %Used
6 0 98304 886 1%
VPN CAM usage: Total Used %Used
512 0 0%
L3 Forwarding Resources
Module FIB TCAM usage: Total Used %Used
6 72 bits (IPv4, MPLS, EoM) 983040 714368 73%
144 bits (IP mcast, IPv6) 32768 24 1%
detail: Protocol Used %Used
IPv4 638057 65%
MPLS 76307 8%
EoM 4 1%
IPv6 17 1%
IPv4 mcast 4 1%
IPv6 mcast 3 1%
Adjacency usage: Total Used %Used
1048576 137937 13%
The breakdown of the above allocation and usage is as follows:
Current TCAM allocation: 16K IPv6 entries + 16k Multicast entries = 960k IPv4 & MPLS entries 1,024,000 * 72 bit FIB TCAM entries == 73,728,000 bit total TCAM size 16,000 * 144 bit IPv6 prefix entries == 32,000 * 72 bit entries = 2,304,000 bits 73,728,000 bit TCAM - 2,304,000 bits for IPv6 = 71,424,000 TCAM bits left 16,000 * 144 bit Multicast prefix entries == 32,000 * 72 bit entries = 2,304,000 bits 71,424,000 bit TCAM - 2,304,000 bits for Multicast = 69,120,000 TCAM bits left 69,120,000 bit TCAM / 72 bits per IPv4 & MPLS entries = 960,000 IPv4 & MPLS FIB size.
When adjusting the FIB TCAM allocation the entries must add up to 1024K entries. By simply setting the IPv4 prefix allocation size all the remaining values are reduced to accommodate automatically. This is an example default configuration:
#show mls cef max FIB TCAM maximum routes : ======================= Current :- ------- IPv4 + MPLS - 512k (default) IPv6 + IP Multicast - 256k (default)
In the below example only “mls cef maximum-routes ip 768” is configured. Notice now that a specific allocation is given for IPv4 prefix FIB size, MPLS has been split off into it's own allocation and is no longer dynamically sharing with IPv4 as above:
#sh mls cef maximum-routes FIB TCAM maximum routes : ======================= Current :- ------- IPv4 - 768k MPLS - 16k (default) IPv6 + IP Multicast - 120k (default)
In the below example only “mls cef maximum-routes ip 850” is configured, note again that MPLS has been split into it's own allocation pool:
#show mls cef maximum-routes FIB TCAM maximum routes : ======================= Current :- ------- IPv4 - 850k MPLS - 14k (default) IPv6 + IP Multicast - 80k (default)
In the below example only “mls cef maximum-routes ip 1000” is configured, note again that MPLS has been split into it's own allocation pool:
# show mls cef maximum-routes FIB TCAM maximum routes : ======================= Current :- ------- IPv4 - 1000k MPLS - 8k (default) IPv6 + IP Multicast - 8k (default)
In the below example “mls cef maximum-routes ipv6 16” and “mls cef maximum-routes ip-multicast 16” are configured, leaving the rest for IPv4 and MPLS. Note that since a specific value hasn't been supplied for either IPv4 or MPLS they are dynamically sharing again and now IPv6 and Mulicast are using their own specific FIB pools:
#show mls cef maximum-routes FIB TCAM maximum routes : ======================= Current :- ------- IPv4 + MPLS - 960k (default) IPv6 - 16k IP Multicast - 16k
If the TCAM allocation configuration is incorrect the following message can be seen (such as if they don’t add up to 1024K 72-bit entries) after reboot:
%MLSCEF-SP-1-MAX_ROUTE_MISMATCH: Maximum routes config mismatch. Reconfigure the maximum routes values and reload the box.
When the TCAM is full the following can be seen:
%MLSCEF-SP-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for MPLS protocol. %C6K_MPLS_LC-SP-3-AGG_LABEL_TCAMFAIL: failed to insert label 25611 (vrf_id 82 hwid 0) to TCAM (err_index -1) -Traceback= 81D7638z 8FD8640z 8FD9818z 8FD9BE0z 8FDDD74z 8FDF200z 8FE038Cz 83F2CF0z 83EDEF0z Or %MLSCEF-SP-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol %MLSCEF-SP-7-FIB_EXCEPTION: FIB TCAM exception, some entries will be software switched
The output of the command "show platform hardware capacity forwarding" can be misleading. Consider the below two examples. In the first one below the router has a shared 960k FIB TCAM entry space for IPv4 and MPLS (and EoMPLS) entries. The "show platform hardware capacity forwarding" command shows 638,057 IPv4 entries and utilisation figure of 65% (638,057 / 983,040 * 100 = 64.90%). For MPLS it shows 76,307 entries as 8% utilisation (76,307 / 983,040 * 100 = 7.76%). The IPv4, MPLS and EoMPLS entries added together are 73% used but it's important to note in the breakdown that for IPv4 the percentage figure is of the total 960K shared entries and the same is true for MPLS, 8% of 960K entries but the total usage is 73%, despite saying 8% for MPLS there isn’t room for another 10x MPLS labels.
#show mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :-
-------
IPv4 + MPLS - 960k (default)
IPv6 - 16k
IP Multicast - 16k
#show platform hardware capacity forwarding
L2 Forwarding Resources
MAC Table usage: Module Collisions Total Used %Used
6 0 98304 886 1%
VPN CAM usage: Total Used %Used
512 0 0%
L3 Forwarding Resources
Module FIB TCAM usage: Total Used %Used
6 72 bits (IPv4, MPLS, EoM) 983040 714368 73%
144 bits (IP mcast, IPv6) 32768 24 1%
detail: Protocol Used %Used
IPv4 638057 65%
MPLS 76307 8%
EoM 4 1%
IPv6 17 1%
IPv4 mcast 4 1%
IPv6 mcast 3 1%
Adjacency usage: Total Used %Used
1048576 137937 13%
In the second example below in the full breakdown from "show platform hardware capacity forwarding" one can see that there are 14,246 MPLS labels in use and the utilisation figure is 2%. However in this case the MPLS allocated FIB TCAM space is actually 100% full. The first command "show mls cef maximum-route" shows that 14k FIB entries are allocated for MPLS labels, in this example only the command “mls cef maximum-routes ip 850” has been configured, split IPv4 and MPLS into separate FIB TCAM allocations. The command "show platform hardware capacity forwarding" is calculating the percentage utilisation figure for MPLS label usage as a percentage of 850k allocated IPv4 FIB entries + 14K allocated MPLS FIB entries, as if they were both still sharing the same space dynamically. 14,246 / 884,736 * 100 = 1.61% (which is rounded up to 2% in the below output):
#show mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :-
-------
IPv4 - 850k
MPLS - 14k (default)
IPv6 + IP Multicast - 80k (default)
#show platform hardware capacity forwarding
L2 Forwarding Resources
MAC Table usage: Module Collisions Total Used %Used
6 0 98304 842 1%
VPN CAM usage: Total Used %Used
512 0 0%
L3 Forwarding Resources
Module FIB TCAM usage: Total Used %Used
6 72 bits (IPv4, MPLS, EoM) 884736 650982 74%
144 bits (IP mcast, IPv6) 81920 24 1%
detail: Protocol Used %Used
IPv4 636736 72%
MPLS 14246 2%
EoM 0 0%
IPv6 17 1%
IPv4 mcast 4 1%
IPv6 mcast 3 1%
Adjacency usage: Total Used %Used
1048576 131068 12% Previous page: 6500/7600 ELAM
Next page: 6500/7600 Forwarding Debug