Date created: Monday, September 9, 2024 8:52:48 AM. Last modified: Friday, October 11, 2024 1:41:37 PM
TCAM Profiles
References
https://www.arista.com/en/support/toi/tcam-profile
https://www.arista.com/en/support/toi/user-defined-tcam-profiles-toi
https://www.arista.com/en/support/toi/eos-4-26-0f/14755-user-defined-tcam-profiles
https://www.arista.com/en/support/toi/eos-4-20-5f/14022-tcam-reservation-on-sand
https://www.arista.com/en/solutions/flexroute-engine-ip-forwarding
https://www.arista.com/en/support/toi/eos-4-27-2f/15092-optimized-ipv4-route-scale-with-8-to-1-compression
https://www.arista.com/en/support/toi/eos-4-21-3f/14164-fib-compression
Routing and Forwarding Scale (TCAM vs MDB Profiles)
Arista forwarding plane resources are split into two sections:
- The Modular Database (MDB), which is not user customisable
- The TCAM profile, which is user customisable
An MDB profile is configured by default and can be changed only to one of the presets in EOS. Users can't create their own MDB profiles. These profiles control the allocation of forwarding resources for base features like IPv4/6 routing, MAC learning ACL entries etc. The profiles allocate a static amount of forwarding resources for these features.
Example using the l3-xxxl MDB profile:
7280R3#show platform sand mdb profile detail
System profile: l3-xxxl
FAP FAP Type Profile
---------- ---------------- --------------
Fap0 Jericho2B1 arista-l3-xxxl
The fixed forwarding resources which are carved up by the MDB profile can be seen in the output of the CLI command "show hardware capacity" (all entries except those starting with "TCAM").
EOS Flex-Route is a configurable option from Arista to increase IP FIB capacity if needed. Flex-Route if required needs to be enabled in the TCAM profile, and uses additional TCAM banks to store forwarding data.
Another option available to increase the FIB size is to enable FIB Compression.
TCAM profiles allows an operator to allocate the limited TCAM resources to optional features, based on the operators feature requirements i.e., no TCAM banks are allocated for features which the operator doesn't need. The TCAM space is dynamically allocated based on the TCAM profile, which is different to the space allocated by the MDB profile which allocates a fixed/static amount of resources.
The TCAM profile states how many resources are required for a specific feature, but the resources are not allocated for that feature until it is actually used. Also, as a feature uses more resources, more TCAM banks will be dynamically allocated for that feature, based on the definition in the TCAM profile.
The configurable forwarding resources which are carved up by defining a custom TCAM profile can be seen in the output of the CLI command "show hardware capacity" (all entries which start with "TCAM").
7280R3 TCAM
The 7280R3 series routers have 16x 160 bit wide TCAM banks. Router features require TCAM space. A bank can be used for a single feature only.
If a feature has to match a source and destination IPv4 address for example, that is 64 bits. That will fit into a single TCAM bank because they are 160 bits wide. Therefore, a single TCAM bank can be allocated for that feature. If a source and destination IPv6 address both have to be matched for a specific feature, that feature will require 2x 160 bit TCAM banks. Most IPv4 features use a single 160 bit TCAM bank and all IPv6 features use two banks.
EOS does support allocating 80 bits to a feature (if say, it only has to match an MPLS label), meaning a TCAM bank can be split and used for two separate features with a smaller number of bits required. EOS allows for 80 bits, 160 bits, or 320 bits to be allocated for a feature (half a bank, 1 bank, or 2 banks), no other sizes, and no custom sizes.
The first 12 TCAM banks store 2048x 160 bit entries, and the last 4 TCAM banks store only 256x 160 bit entries. However, allocating 1x 160 bit TCAM bank for an IPv4 feature for example, doesn't guarantee 2048 entries for that feature. Equally, allocating 2x 160 bit banks for an IPv6 feature doesn't guarantee 2048 entries for that feature. The reason for this is that features which use TCAM space can use two different methods to squeeze data into the TCAM, they can spread data over multiple entries either "vertical scaling" or multiple banks "horizontal scaling" (not official Arista terms!).
For example, let's consider a single IPv4 FlowSpec rule. If all the data required for matching packet headers fits into 160-bits, then it would use a single TCAM entry in a single TCAM bank. Let's pretend that a matched packet may have one of four possible actions applied to it, and only two actions can be encoded in a single TCAM entry. In this case, the same FlowSpec rule will be programmed twice, using two entries in a TCAM bank, one entry maps to actions 1 & 2, the other entry maps to actions 3 & 4. This is "vertical" scaling (using multiple rows/entries in a bank). As a result a single FlowSpec rule has used two TCAM entries. "horizontal" scaling then, would be the case where 160-bits isn't enough space to match all the packet header options and two TCAM banks have to be used to store up to 320-bits of header matching data. This results in a single FlowSpec rule being split across an entry in TCAM bank 1 and an entry in TCAM bank 2. In this case we can still get 2048 entries per bank, but we're using twice as many banks. And just for fun, TCAM features can employ both horizontal and vertically scaling techniques at the same time, if the feature is sufficiently complex and needs to store a lot of data. In this case two entries are used in each TCAM bank meaning four TCAM entries in total. An example of this can be seen in the output "show platform fap acl tcam hw bank" when complex features like Flowspec are configured and complex rules have been configured (that match a range of ports for example).
When a TCAM bank becomes full, if there are free unallocated TCAM banks, a free bank will be allocated to a feature as it requests additional TCAM entires. TCAM profiles describe how TCAM banks should be allocated when they are needed. TCAM banks are allocated and released dynamically as usage increases and decreases. This means that TCAM profiles can oversubscribe TCAM resources, because usage is demand based. Even when 320-bits of TCAM space are allocated for a feature in a TCAM profile, if the feature only needs 160-bits of space, then only one TCAM bank will be used by that feature. This means it's quite difficult to tell from a TCAM profile, how much TCAM space the features on a router will use.
In addition to the 16x TCAM banks used for matching traffic, there are 32x TCAM banks used for then storing the action to be taken against the matched traffic. The first 24 action banks store 2048x 32-bit entries. The last 8 action banks store 256x 32-bit entries.
When a key size of 80 bits is allocated for a feature, 1x action bank is allocated for the feature to store the actions for that traffic. When a key size of 160 bits is allocate for a feature, 2x action banks are allocated for that feature (giving 64 bits for the action). When a key size of 320 bits is allocated for a feature, 4x action banks are allocated for that feature (giving 128 bits for the action). This means that some features may need to be configured to use 320-bits/2 Banks, because they require more actions (even though the key fields fit in 80 or 160 bits of space).
Key Size Limit
Note that when configuring TCAM features on EOS, by default 2x 160bit TCAM banks are allocated for the feature. This is fine for IPv6. For IPv4 only 1x 160bit TCAM bank is needed (most of the time, sometimes only 80 bits are needed). Therefore, IPv4 features are often explicitly configured with key size limit 160. This stops a feature from using more TCAM banks than expected, but if that extra space was needed, it could stop the feature from working at all. Without "key size limit 160" 2x 160bit TCAM banks can be allocated for an IPv4 feature, but will only be allocated if needed. This is also why it is best practice to specify no key size for IPv6 features, because they default to 320 bits.
Priority
TCAM features have a configurable priority. If Flowspec is active on an interface and a traffic policy, and they both have rules in TCAM which match an incoming packet, one needs to be chosen to decide which action to take. If any feature has a drop action, this take absolute priority, if there are no drop actions, then the action from the highest matching feature (i.e., Flowspec or Traffic-Policy or Policer or ACL) is performed.
Below the priority of all TCAM features can be see, the features which are enabled in the running TCAM policy are listed at the top, unused features are listed below in priority order (here, the default priority of the configured features hasn't been changed):
7280R3#show platform fap tcam action-priority
TCAM Database Action Priority (Fap0)
Database Active Default
----------------------------------------------- -------------- --------
dbEgressIpAcl 805 805
dbTrafficPolicyIp6Egress 515 515
dbTrafficPolicyIp4Egress 510 510
dbDirectExtractionPmf3 505 505
dbOamNoSysHeader 495 495
dbTrafficPolicyIp6 200 200
dbTrafficPolicyIp4 195 195
dbEvpnMplsEsiFiltering 190 190
dbL2ProtocolFwd 185 185
dbEvpnMplsEsiLocalToRemote 180 180
dbDirectExtractionPmf1 170 170
dbMplsFemDirectExtraction 165 165
dbFlowspecIp6 80 80
dbEvpnMplsBridging 51 51
dbTunnelTermination 45 45
dbFlowspecIp 40 40
dbVxlanVarp 35 35
dbMplsSystem 25 25
dbIpMirror 15 15
-------- -------- --------
dbEgressIp6Racl - 830
dbEgressIp6Pacl - 825
dbEgressVxlanDrop - 820
dbEgressRfc2544 - 815
dbEgressMacAcl - 810
dbEgressMirrorCongestion - 800
dbOam - 495
dbSegmentSecurityIp6 - 235
dbSegmentSecurityIp4 - 230
dbIp6PostcardPolicy - 225
dbIp4SampledFlowTracking - 220
dbIpPostcardPolicy - 215
dbIp6CpuTrafficPolicy - 210
dbIpCpuTrafficPolicy - 205
dbSflowFilteredSampling - 205
dbTrafficPolicyIp6VlanInterface - 194
dbTrafficPolicyIp4VlanInterface - 193
dbFlow - 175
dbDirectExtractionPmf1Srv6 - 170
dbDirectExtractionPmf3Srv6 - 169
dbIpTcpMss - 166
dbHashingReplacementIpv4 - 164
dbHashingReplacementIpv4Null - 163
dbHashingReplacementIpv6High - 162
dbHashingReplacementIpv6Low - 161
dbMacQos - 160
dbMacMirror - 155
dbTapTunnelTermination - 150
dbIp6TapAgg - 145
dbIpTapAgg - 140
dbMacTapAgg - 135
dbVniLookup - 131
dbIpTunnelIntfPbr - 131
dbMplsPbr - 130
dbIp6Pbr - 125
dbIpPbr - 120
dbIp6Qos - 115
dbEgressIp6SharedAcl - 110
dbTapPort - 105
dbIp6Mirror - 100
dbIpTunnelIntfAcl - 95
dbIpPacl - 85
dbCbf - 75
dbIpRacl - 70
dbIp6Pacl - 65
dbIp6Racl - 60
dbIpQos - 50
dbVxlanMultihomingMcast - 45
dbVxlanFemDirectExtraction - 45
dbVxlanVarp6 - 36
dbMacPacl - 30
dbIp6VrfSelection - 1
dbPvlan - 1
dbUrpfLookupVrf - 1
dbIpv4TtBasic - 1
dbDirectExtractionPmf1StormControlSubIntf - 0
dbDirectExtractionSetDscp - 0
dbDirectExtractionSetUnderlaySysInLif - 0
dbDirectExtractionPmf1GroupPolicer - 0
dbDirectExtractionPmf1IntfPolicingCounter - 0
dbDirectExtractionSetEcn - 0
dbSecondaryVrfDestDirectExtraction - 0
dbDirectExtractionPmf1SubintfSflow - 0
TCAM Usage
The TCAM profile states how many resources are required for a specific feature, but the resources are not allocated for that feature until it is actually used. Also, as a feature uses more resources, more TCAM banks will be dynamically allocated for that feature, based on the definition in the TCAM profile.
Example output is shown below from a lab 7280R3. It can be seen that 28 TCAM banks worth of features are configured in the profile:
7280R3#show hardware tcam profile XXXX detail | inc Key size Key size: 160 Key size: 160 Key size: 320 Key size: 160 Key size: 160 Key size: 160 Key size: 320 Key size: 320 Key size: 320 Key size: 160 Key size: 320 Key size: 320 Key size: 160 Key size: 320 Key size: 320 Key size: 320 Key size: 160 Key size: 320
However, the device only has 16 TCAM banks.
One can see which banks are used in the command output below:
7280R3#show platform fap tcam summary
Tcam Allocation (Fap0)
Bank Used By Reserved By
---------- -------------------------------- -----------
0 dbEvpnMplsEsiFiltering -
1 dbEvpnMplsEsiLocalToRemote -
1 dbEvpnMplsBridging -
2 dbIpv6MpTtBasic -
3 dbTrafficPolicyIp4 -
3 dbTrafficPolicyIp6 -
4 dbTrafficPolicyIp4Egress -
4 dbTrafficPolicyIp6Egress -
5 dbL2ProtocolFwd -
12 dbSystem -
12 dbSystem6 -
12 dbMplsSystem -
13 dbEgressOlp -
15 dbMacSystem -
15 dbTunnelTermination -
Banks 0, 1, 2, 3, 4, 5, 12, 13, and 15 are used, meaning banks 6, 7, 8, 9, 10, 11, and 14 are free. One can also see that some features are sharing a TCAM banks.
Previous page: 7280R3 Forwarding Scale and Utilisation
Next page: Local Route Leaking Agent