Date created: Wednesday, October 19, 2016 4:07:18 PM. Last modified: Thursday, July 16, 2020 2:58:14 PM
6500/7600 Forwarding Hardware
References:
Cisco Expo 2008 - Cisco 7600 Architecture by Aleksandar Vidakovic
BRKARC-2350 - IOS Routing Internals (2013)
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/116643-technote-product-00.html
http://www.cisco.com/c/en/us/td/docs/ios/vswitch/command/reference/vs_book/vs_02.pdf
http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-6500-series-switches/white_paper_c11-652042.html
https://wenku.baidu.com/view/596b34a9d1f34693daef3e8d.html
Contents:
7600 Hardware Overview
7600 Card ASICs
Protocol RIBs feed into the central RIB (with administrative distance acting as the tie breaker between multiple protocols offering the same route into the RIB) and then the RIB with the egress interface destination and L2 rewrite information are compiled into the FIB (in software). In hardware accelerated platforms (separate CPU for control place and hardware for the data plane) like the 6500/7600, Nexus 7000 or ASR9000 etc the FIB is then programmed into the hardware (typically TCAM).
In routers like the 7600 the various parts of the forwarding are distributed amongst the various removable components within the chassis (the RSP, line cards and DFCs).
Below the PFC on the RSP contains the FIB TCAM for forwarding, QoS TCAM and ACL TCAM. Packets that ingress a CFC line card query the TCAM on the PFC with the packet header used as the search query. The request is sent over the DBUS (data bus) and the result (egress port, header rewrite info etc) is returned over the RBUS (result bus). The ingress line card then packets the packet into a superframe and transmits it across the backplane fabric to the egress line card. For DFC enabled line cards a copy of the TCAM is placed directly on the line card so the egress lookup can happen on the packet ingressing line card.
Above the MSFC is shown with the separate SP and RP that provide the L2 and L3 control planes. They both use the EOBC channel to communicate the PFC TCAMs. The MSFC builds the Cisco Express Forwarding information Base (FIB) table in software and then downloads this table to the hardware ASICs on the PFC and DFCs (if present) that make the forwarding decisions for IP unicast and multicast traffic.
The PFC3/DFC3 generation is built upon a forwarding architecture known as EARL7. ES+ line cards use EARL7.5
On the PFC3/DFC3 (EARL7) the Superman ASIC is the L2 forwarding engine and the Tycho ASIC is the L3 engine on the PFC.
On the RSP720/PFC3C (EARL 7.5) the Superman and Tycho ASICs are squahsed into one called the Supertycho and another ASIC called Kuma is present providing the lookup bus functionality.
Below are the channel layout for a 7609-S and 7613 (for a 7606-S it is the same as the 7609-S, dual 20Gbps channels per slot), lots more info here: /index.php?page=6500-7600-chassis-architecture
Below is the line card architecture for a classic LAN card using CFC and DFC:
Below are some outputs from an example 7606-S router with LAN cards all using CFCs and an RSP720-3CXL-10GE:
abr1#show module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX XXXXXXXXXX
2 48 CEF720 48 port 1000mb SFP WS-X6748-SFP XXXXXXXXXX
3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX XXXXXXXXXX
4 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE XXXXXXXXXX
5 5 Route Switch Processor 720 10GE (Activ RSP720-3CXL-10GE XXXXXXXXXX
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ----- ------------- ------------ -------
1 30e4.db0f.0928 to 30e4.db0f.0957 4.1 12.2(18r)S 15.3(3)S Ok
2 e8b7.48ed.60c4 to e8b7.48ed.60f3 2.4 12.2(18r)S 15.3(3)S Ok
3 4c00.82fe.7ad0 to 4c00.82fe.7aff 4.3 12.2(18r)S 15.3(3)S Ok
4 44d3.ca80.f16c to 44d3.ca80.f16f 3.2 12.2(14r)S 15.3(3)S Ok
5 001c.584e.e8e4 to 001c.584e.e8eb 2.2 12.2(33r)SRC 15.3(3)S Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Centralized Forwarding Card WS-F6700-CFC XXXXXXXXXX 4.1 Ok
2 Centralized Forwarding Card WS-F6700-CFC XXXXXXXXXX 4.1 Ok
3 Centralized Forwarding Card WS-F6700-CFC XXXXXXXXXX 4.1 Ok
4 Centralized Forwarding Card WS-F6700-CFC XXXXXXXXXX 4.1 Ok
5 Policy Feature Card 3 7600-PFC3CXL-10GE XXXXXXXXXX 1.2 Ok
5 C7600 MSFC4 Daughterboard 7600-MSFC4 XXXXXXXXXX 2.1 Ok
abr1#show platform hardware pfc mode
PFC operating mode : PFC3CXL
abr1#sho fabric switching-mode
Global switching mode is Compact
dCEF mode is not enforced for system to operate
Fabric module is not required for system to operate
Modules are allowed to operate in bus mode
Truncated mode is allowed, due to presence of DFC, CEF720 module
Module Slot Switching Mode
1 Crossbar
2 Crossbar
3 Crossbar
4 Crossbar
5 dCEF
abr1#show fabric utilization all
slot channel speed Ingress % Egress %
1 0 20G 0 0
1 1 20G 0 0
2 0 20G 0 1
2 1 20G 0 0
3 0 20G 0 0
3 1 20G 0 0
4 0 20G 20 4
4 1 20G 4 11
5 0 20G 0 8
5 1 20G 0 0
abr1#show platform hardware asic-versions
ASIC Name Version
slot : 1
GEMINI (55) (3.0)
JANUS (48) (1.0)
SSA (43) (9.0)
ROHINI (49) (1.5)
slot : 2
GEMINI (55) (3.0)
JANUS (48) (1.0)
SSA (43) (9.0)
ROHINI (49) (1.6)
slot : 3
GEMINI (55) (3.0)
JANUS (48) (1.0)
SSA (43) (9.0)
ROHINI (49) (1.5)
slot : 4
GEMINI (55) (3.0)
JANUS (48) (1.0)
SSA (43) (9.0)
ROHINI (49) (1.6)
slot : 5
ST_TYCHO (68) (3.2)
ST_SUPERMAN (67) (2.2)
KUMA (70) (3.0)
METRO_ARGOS (73) (3.0)
METRO_KRYPTON (72) (3.0)
SSA (43) (9.0)
SANTA_CRUZ (76) (3.0)
TELESTO (82) (7.0)
KAILASH (83) (2.4)
R2D2 (69) (3.0)
slot : 6
slot : 7
slot : 8
slot : 9
slot : 10
slot : 11
slot : 12
slot : 13
From: http://www.cisco.com/c/en/us/td/docs/ios/vswitch/command/reference/vs_book/vs_02.pdf Lyra — Layer 2 forwarding engine Hyperion — Packet rewrite, multicast, and SPAN engine Medusa — Crossbar and bus fabric interface Polaris — Layer 3 CEF engine Pinnacle — 4-port Gigabit Ethernet interface Titan — Packet rewrite and replication engine Vela — Bus interface Kuma 2 — Bus bridge/converter ASIC. Metro_Argos 2 — Metropolis Argos ASIC. Bridge between the port, fabric, and the forwarding engine. Metro_Krypton 2 —Metropolis Krypton ASIC. Interface ASIC that sits between a port ASIC and the EARL complex. SSA 2 — Super Santa Anna ASIC. Serial link transceiver back to fabric. R2D2 4 — Port interface ASIC. SSA 1 — Super Santa Anna ASIC. Serial link transceiver back to fabric. SSO 4 — Crossbar switch fabric ASIC. Tiangang 4 — Interface between two R2D2s to be interfaced to one channel of the Metropolis ASIC instead of one R2D2 per channel
From: https://wenku.baidu.com/view/596b34a9d1f34693daef3e8d.html
Alkindi = 1G PHY ASIC (with MAC SEC / Cisco Trusted Security)
Ganita = 10G PHY ASIC (with MAC SEC / Cisco Trusted Security)
Eureka and Lamira ASICs = EARL8 chipset.
Ringer ASIC = VLCC (Virtua Linecard Console/RS32) chip on WS-X6908-10G
Estelle ASIC = VLCC (Virtua Linecard Console/RS32) chip on WS-X6904-40G
In the WS-X6724-SFP, WS-X6748-SFP and WS-X6748-GETX a single "port" ASIC supports 12 local Gigabit Ethernet ports. In the WS-X6704-10GE line card, there are four port ASICs each supporting 1 x 10 Gigabit Ethernet ports. For WS-X6708-10GE cards there 8 port ASICs, each supporting 1x 10 Gigabit Ethernet ports. For the 48x1G card and 4x10G card the port ASIC is the ROHINI ASIC. On the 8x10G card it is the R2D2 ASIC. Two ROHINI's feed into one JANUS and each JANUS feeds into an SSA ASIC ("Super Santa Ana" which are 20Gbps switch fabric channel ASICs).
For the WS-X6708-10GE there are 8x R2D2 port ASICs (ASIC per port) -> 4 Tiangang aggregation ASICs (port pair ASIC which aggregates 2x R2D2s into a single Metropolis ASIC channel instead of 1x R2D2 per Metropolis channel) -> Metro Argos (Metropolis Argos ASIC, bridge between the port, fabric, and the forwarding engine) -> 2 SSA ASICS (Super Santa Anna, fabric complex ASIC). The Metro Krypton ASIC (Metropolis Krypton) is an interface that sits between a port ASIC and the EARL complex and can handle Multicast. The Kuma ASIC handles the forwarding bus lookup.
For the WS-X6716-10G two ports share a single 10G port ASIC (the NAXOS ASIC) and two port pairs (two NAXOS ASICs) feed into a single ASHWINI ASIC which is also 10Gbps. These feed into 4x R2D2 then 2x Metro Argos then 2x Super Santa Ana's.
There is lots of info on 6500/7600 line card ASICs here: /index.php?page=6500-7600-qos#ports-asics-fabric
The 48x1G cards and 4x10G card have the same number of each ASIC (JANUS, SSA and ROHINI):
7600#show asic-version slot 1
Module in slot 1 has 3 type(s) of ASICs ! WS-X6748-GE-TX
ASIC Name Count Version
JANUS 2 (1.0)
SSA 2 (9.0)
ROHINI 4 (1.5)
7600#show asic-version slot 2
Module in slot 2 has 3 type(s) of ASICs ! WS-X6748-SFP
ASIC Name Count Version
JANUS 2 (1.0)
SSA 2 (9.0)
ROHINI 4 (1.6)
7600#show asic-version slot 3
Module in slot 3 has 3 type(s) of ASICs ! WS-X6704-10GE
ASIC Name Count Version
JANUS 2 (1.0)
SSA 2 (9.0)
ROHINI 4 (1.6)
7600#show asic-version slot 5
Module in slot 5 has 8 type(s) of ASICs ! WS-X6708-10GE
ASIC Name Count Version
KUMA 1 (3.0)
METRO_ARGOS 1 (3.0)
METRO_KRYPTON 1 (3.0)
SSA 2 (9.0)
SANTA_CRUZ 1 (3.0)
TELESTO 1 (7.0)
KAILASH 1 (2.4)
R2D2 2 (3.0)
#Outputs from WS-X6708-10G show asic slot ASIC Name Count Version KUMA 2 (3.0) METRO_ARGOS 2 (3.0) METRO_KRYPTON 2 (3.0) SSA 2 (9.0) R2D2 8 (2.0) TIANGANG 4 (54.0) Te/1 Ports-in-ASIC (Sub-port ASIC) : 1,4-5,7 (1) Te/2 Ports-in-ASIC (Sub-port ASIC) : 2-3,6,8 (2) # Outputs from WS-X6716-10G #show asic slot 1 Module in slot 1 has 7 type(s) of ASICs ASIC Name Count Version KUMA 2 (3.0) METRO_ARGOS 2 (3.0) METRO_KRYPTON 2 (3.0) SSA 2 (9.0) NAXOS 8 (1.0) R2D2 4 (3.0) ASHWINI 4 (0.6) #show interface Te1/1 capabilities | inc ASIC Ports-in-ASIC (Sub-port ASIC) : 1-8 (1-4) #show interface Te1/5 capabilities | inc ASIC Ports-in-ASIC (Sub-port ASIC) : 1-8 (5-8) #show interface Te1/9 capabilities | inc ASIC Ports-in-ASIC (Sub-port ASIC) : 9-16 (9-12) #show interface Te1/13 capabilities | inc ASIC Ports-in-ASIC (Sub-port ASIC) : 9-16 (13-16)
For the WS-X6908-10G each port has a Ganita ASIC which is the PHY ASIC, each GANITA feeds into one R2D2 port ASIC (10G ASIC). The ports are paired 1&2, 3&4 etc, and feed into port-group ASICs (Metro Argos). Metro Argos > SSA (Super Santa Anna) ASICs provide the fabric (serial) connection > feeds into the Santa Monica ASIC providing the backplane connectivity.
! WS-X6908-10G
6509E-SUPT2T#show asic-version switch 1 slot 1
Module in switch 1, slot 1 has 7 type(s) of ASICs ! DCEF-2T WS-X6908-10G
ASIC Name Count Version
METRO_ARGOS 4 (4.0)
METRO_KRYPTON 4 (4.0)
MALFRA 2 (16.4)
SSA 4 (9.0)
SANTA_MONICA 1 (1.0)
GANITA 8 (2.0)
R2D2 8 (5.0)
Ports-in-ASIC (Sub-port ASIC) : 1-2 (1) Ports-in-ASIC (Sub-port ASIC) : 3-4 (3) Ports-in-ASIC (Sub-port ASIC) : 5-6 (5) Ports-in-ASIC (Sub-port ASIC) : 7-8 (7)
7600#show asic-version slot 1
Module in slot 1 has 5 type(s) of ASICs ! 7600-ES20-10G3CXL
ASIC Name Count Version
SSA 1 (0.0)
KRYPTON 1 (14.0)
BLACKWATER 1 (3.0)
XCHIP 1 (2.4)
RASP16 1 (2.2)
7600-sp#show platform hardware asic-versions
ASIC Name Version
slot : 1
ST_TYCHO (68) (3.2)
ST_SUPERMAN (67) (2.2)
BLACKWATER (62) (3.0)
XCHIP (63) (2.4)
RASP16 (64) (2.2)
KRYPTON (52) (14.0)
KUMA (70) (3.0)
SSA (43) (0.0)
Below are the card and ASIC layouts for the ES20 and ES40 cards:
From the various supervisor line cards the SANTA_CRUZ and SANTA_MONICA ASICs are some how related to the forwarding (they provide the backplane connectivity). The Trident line cards in the ASR9000 series are based on the NP3C network processing unit used in ES+ line cards (so they are basically beefed up ES+ line cards) and use the SANTA_CRUZ ASIC. The Santa Cruz ASIC was replaced by the Santa Monica ASIC. The Santa Cruz was 20 ports, 20Gbps max each == 360Gbps total. The Santa Monica suports 26 ports, 40Gbps max each == 1040Gbps.
ES20 line cards used the PXF (Parallel eXpress Forwarding) engine codenamed "Toaster".
On the SUP720 and RSP720 (with EARL7) the Superman ASIC performs the layer 2 forwarding lookup operations and the Tycho ASIC performs the layer 3 forwarding lookup operations. On the RSP720-3C/DFC3C EARL 7.5 is used, Superman and Tycho are one single ASIC called Supertycho so when performing an ELAM against the Superman ASIC Supertycho is presumably called as this seems to capture layer 2, MPLS and native layer 3 traffic. The link between Superman/Tycho/Supertycho and the SANTA_CRUZ / SANTA_MONICA ASICs is not clear. Also the KAILASH ASIC is actually an FPGA which has something to do with parsing RP and SP CPU punted/originated traffic. The TELESTO ASIC is something to do with fabric connectivity?
For SUP2T the Eureka and Lamira ASICs form the EARL8 chip set and the layer 2 forwarding engine, Eureka is used for ELAM captures on SUP2T/PFC4/DFC4. Below are outputs from a few 7600s and 6500s showing each supervisor has one of either the SANTA_MONICA or SANTA_CRUZ ASIC.
!
! 6509-E VSS stack with VS-SUP2T-10G
!
6509-E#show module switch 1 slot 5
Switch Number: 1 Role: Virtual Switch Active
---------------------- -----------------------------
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
5 5 Supervisor Engine 2T 10GE w/ CTS (Acti VS-SUP2T-10G XXXXXXXXXXX
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
5 0006.f614.cc72 to 0006.f614.cc79 1.5 12.2(50r)SYS 15.1(1)SY1 Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
5 Policy Feature Card 4 VS-F6K-PFC4 XXXXXXXXXXX 2.1 Ok
5 CPU Daughterboard VS-F6K-MSFC5 XXXXXXXXXXX 2.1 Ok
Mod Online Diag Status
---- -------------------
5 Pass
6509-E#show asic-version switch 1 slot 5
Module in switch 1, slot 5 has 10 type(s) of ASICs
ASIC Name Count Version
KUMA 1 (3.0)
METRO_ARGOS 1 (4.0)
METRO_KRYPTON 1 (4.0)
SSA 2 (9.0)
SANTA_MONICA 1 (1.0)
TELESTO 1 (8.0)
KAILASH 1 (4.0)
GANITA 2 (2.0)
R2D2 2 (5.0)
ALKINDI 1 (4.127)
!
! 7606-S with RSP720-3CXL-10G
!
7606-S#show module 5
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
5 5 Route Switch Processor 720 10GE (Activ RSP720-3CXL-10GE XXXXXXXXXXX
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ----- ------------- ------------ -------
5 001c.584e.e8e4 to 001c.584e.e8eb 2.2 12.2(33r)SRC 15.3(3)S Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
5 Policy Feature Card 3 7600-PFC3CXL-10GE XXXXXXXXXXX 1.2 Ok
5 C7600 MSFC4 Daughterboard 7600-MSFC4 XXXXXXXXXXX 2.1 Ok
Mod Online Diag Status
---- -------------------
5 Pass
7606-S#show asic-version slot 5
Module in slot 5 has 8 type(s) of ASICs
ASIC Name Count Version
KUMA 1 (3.0)
METRO_ARGOS 1 (3.0)
METRO_KRYPTON 1 (3.0)
SSA 2 (9.0)
SANTA_CRUZ 1 (3.0)
TELESTO 1 (7.0)
KAILASH 1 (2.4)
R2D2 2 (3.0)
!
! 6513 (non-E) with SUP720
!
6513#show module 7
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
7 2 Supervisor Engine 720 (Active) WS-SUP720-3B XXXXXXXXXXX
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
7 0016.4708.3314 to 0016.4708.3317 5.3 8.4(2) 12.2(33)SXI9 Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
7 Policy Feature Card 3 WS-F6K-PFC3B XXXXXXXXXXX 2.3 Ok
7 MSFC3 Daughterboard WS-SUP720 XXXXXXXXXXX 2.6 Ok
Mod Online Diag Status
---- -------------------
7 Pass
6513#show asic-version slot 7
Module in slot 7 has 4 type(s) of ASICs
ASIC Name Count Version
SSA 1 (9.0)
SSO 4 (2.0)
HYPERION 1 (6.0)
PINNACLE 1 (4.0)
Previous page: 6500/7600 Forwarding Debug
Next page: 6500/7600 NetDR Captures