Date created: Wednesday, October 19, 2016 4:07:18 PM. Last modified: Thursday, July 16, 2020 2:58:14 PM

6500/7600 Forwarding Hardware

References:
Cisco Expo 2008 - Cisco 7600 Architecture by Aleksandar Vidakovic
BRKARC-2350 - IOS Routing Internals (2013)
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/116643-technote-product-00.html
http://www.cisco.com/c/en/us/td/docs/ios/vswitch/command/reference/vs_book/vs_02.pdf
http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-6500-series-switches/white_paper_c11-652042.html
https://wenku.baidu.com/view/596b34a9d1f34693daef3e8d.html

Contents:
7600 Hardware Overview
7600 Card ASICs

7600 Hardware Overview

Protocol RIBs feed into the central RIB (with administrative distance acting as the tie breaker between multiple protocols offering the same route into the RIB) and then the RIB with the egress interface destination and L2 rewrite information are compiled into the FIB (in software). In hardware accelerated platforms (separate CPU for control place and hardware for the data plane) like the 6500/7600, Nexus 7000 or ASR9000 etc the FIB is then programmed into the hardware (typically TCAM).

 

In routers like the 7600 the various parts of the forwarding are distributed amongst the various removable components within the chassis (the RSP, line cards and DFCs).

Below the PFC on the RSP contains the FIB TCAM for forwarding, QoS TCAM and ACL TCAM. Packets that ingress a CFC line card query the TCAM on the PFC with the packet header used as the search query. The request is sent over the DBUS (data bus) and the result (egress port, header rewrite info etc) is returned over the RBUS (result bus). The ingress line card then packets the packet into a superframe and transmits it across the backplane fabric to the egress line card. For DFC enabled line cards a copy of the TCAM is placed directly on the line card so the egress lookup can happen on the packet ingressing line card.

Above the MSFC is shown with the separate SP and RP that provide the L2 and L3 control planes. They both use the EOBC channel to communicate the PFC TCAMs. The MSFC builds the Cisco Express Forwarding information Base (FIB) table in software and then downloads this table to the hardware ASICs on the PFC and DFCs (if present) that make the forwarding decisions for IP unicast and multicast traffic.

The PFC3/DFC3 generation is built upon a forwarding architecture known as EARL7. ES+ line cards use EARL7.5

On the PFC3/DFC3 (EARL7) the Superman ASIC is the L2 forwarding engine and the Tycho ASIC is the L3 engine on the PFC.

On the RSP720/PFC3C (EARL 7.5) the Superman and Tycho ASICs are squahsed into one called the Supertycho and another ASIC called Kuma is present providing the lookup bus functionality.

Below are the channel layout for a 7609-S and 7613 (for a 7606-S it is the same as the 7609-S, dual 20Gbps channels per slot), lots more info here: /index.php?page=6500-7600-chassis-architecture

Below is the line card architecture for a classic LAN card using CFC and DFC:

 

Below are some outputs from an example 7606-S router with LAN cards all using CFCs and an RSP720-3CXL-10GE:


abr1#show module
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     XXXXXXXXXX
  2   48  CEF720 48 port 1000mb SFP              WS-X6748-SFP       XXXXXXXXXX
  3   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     XXXXXXXXXX
  4    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      XXXXXXXXXX
  5    5  Route Switch Processor 720 10GE (Activ RSP720-3CXL-10GE   XXXXXXXXXX

Mod MAC addresses                       Hw   Fw            Sw           Status
--- ---------------------------------- ----- ------------- ------------ -------
  1  30e4.db0f.0928 to 30e4.db0f.0957  4.1   12.2(18r)S    15.3(3)S     Ok
  2  e8b7.48ed.60c4 to e8b7.48ed.60f3  2.4   12.2(18r)S    15.3(3)S     Ok
  3  4c00.82fe.7ad0 to 4c00.82fe.7aff  4.3   12.2(18r)S    15.3(3)S     Ok
  4  44d3.ca80.f16c to 44d3.ca80.f16f  3.2   12.2(14r)S    15.3(3)S     Ok
  5  001c.584e.e8e4 to 001c.584e.e8eb  2.2   12.2(33r)SRC  15.3(3)S     Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  1  Centralized Forwarding Card WS-F6700-CFC       XXXXXXXXXX 4.1    Ok
  2  Centralized Forwarding Card WS-F6700-CFC       XXXXXXXXXX 4.1    Ok
  3  Centralized Forwarding Card WS-F6700-CFC       XXXXXXXXXX 4.1    Ok
  4  Centralized Forwarding Card WS-F6700-CFC       XXXXXXXXXX 4.1    Ok
  5  Policy Feature Card 3       7600-PFC3CXL-10GE  XXXXXXXXXX 1.2    Ok
  5  C7600 MSFC4 Daughterboard   7600-MSFC4         XXXXXXXXXX 2.1    Ok

abr1#show platform hardware pfc mode
PFC operating mode : PFC3CXL

abr1#sho fabric switching-mode
Global switching mode is Compact
dCEF mode is not enforced for system to operate
Fabric module is not  required for system to operate
Modules are allowed to operate in bus mode
Truncated mode is allowed, due to presence of DFC, CEF720 module

Module Slot     Switching Mode
    1                 Crossbar
    2                 Crossbar
    3                 Crossbar
    4                 Crossbar
    5                     dCEF

abr1#show fabric utilization all
 slot    channel      speed    Ingress %     Egress %
    1          0        20G            0            0
    1          1        20G            0            0
    2          0        20G            0            1
    2          1        20G            0            0
    3          0        20G            0            0
    3          1        20G            0            0
    4          0        20G           20            4
    4          1        20G            4           11
    5          0        20G            0            8
    5          1        20G            0            0

abr1#show platform hardware asic-versions
            ASIC Name      Version
slot : 1
           GEMINI (55)     (3.0)
            JANUS (48)     (1.0)
              SSA (43)     (9.0)
           ROHINI (49)     (1.5)
slot : 2
           GEMINI (55)     (3.0)
            JANUS (48)     (1.0)
              SSA (43)     (9.0)
           ROHINI (49)     (1.6)
slot : 3
           GEMINI (55)     (3.0)
            JANUS (48)     (1.0)
              SSA (43)     (9.0)
           ROHINI (49)     (1.5)
slot : 4
           GEMINI (55)     (3.0)
            JANUS (48)     (1.0)
              SSA (43)     (9.0)
           ROHINI (49)     (1.6)
slot : 5
         ST_TYCHO (68)     (3.2)
      ST_SUPERMAN (67)     (2.2)
             KUMA (70)     (3.0)
      METRO_ARGOS (73)     (3.0)
    METRO_KRYPTON (72)     (3.0)
              SSA (43)     (9.0)
       SANTA_CRUZ (76)     (3.0)
          TELESTO (82)     (7.0)
          KAILASH (83)     (2.4)
             R2D2 (69)     (3.0)
slot : 6
slot : 7
slot : 8
slot : 9
slot : 10
slot : 11
slot : 12
slot : 13

 

7600 Card ASICs

From: http://www.cisco.com/c/en/us/td/docs/ios/vswitch/command/reference/vs_book/vs_02.pdf

Lyra — Layer 2 forwarding engine 
Hyperion — Packet rewrite, multicast, and SPAN engine
Medusa — Crossbar and bus fabric interface
Polaris — Layer 3 CEF engine 
Pinnacle — 4-port Gigabit Ethernet interface 
Titan — Packet rewrite and replication engine
Vela — Bus interface 
Kuma 2 — Bus bridge/converter ASIC. 
Metro_Argos 2 — Metropolis Argos ASIC. Bridge between the port, fabric, and the forwarding engine.
Metro_Krypton 2 —Metropolis Krypton ASIC. Interface ASIC that sits between a port ASIC and the EARL complex. 
SSA 2 — Super Santa Anna ASIC. Serial link transceiver back to fabric. 
R2D2 4 — Port interface ASIC. 
SSA 1 — Super Santa Anna ASIC. Serial link transceiver back to fabric. 
SSO 4 — Crossbar switch fabric ASIC. 
Tiangang 4 — Interface between two R2D2s to be interfaced to one channel of the Metropolis ASIC instead of one R2D2 per channel

From: https://wenku.baidu.com/view/596b34a9d1f34693daef3e8d.html
Alkindi = 1G PHY ASIC (with MAC SEC / Cisco Trusted Security)
Ganita = 10G PHY ASIC (with MAC SEC / Cisco Trusted Security)
Eureka and Lamira ASICs = EARL8 chipset.
Ringer ASIC = VLCC (Virtua Linecard Console/RS32) chip on WS-X6908-10G
Estelle ASIC = VLCC (Virtua Linecard Console/RS32) chip on WS-X6904-40G

In the WS-X6724-SFP, WS-X6748-SFP and WS-X6748-GETX a single "port" ASIC supports 12 local Gigabit Ethernet ports. In the WS-X6704-10GE line card, there are four port ASICs each supporting 1 x 10 Gigabit Ethernet ports. For WS-X6708-10GE cards there 8 port ASICs, each supporting 1x 10 Gigabit Ethernet ports. For the 48x1G card and 4x10G card the port ASIC is the ROHINI ASIC. On the 8x10G card it is the R2D2 ASIC. Two ROHINI's feed into one JANUS and each JANUS feeds into an SSA ASIC ("Super Santa Ana" which are 20Gbps switch fabric channel ASICs).

For the WS-X6708-10GE there are 8x R2D2 port ASICs (ASIC per port) -> 4 Tiangang aggregation ASICs (port pair ASIC which aggregates 2x R2D2s into a single Metropolis ASIC channel instead of 1x R2D2 per Metropolis channel) -> Metro Argos (Metropolis Argos ASIC, bridge between the port, fabric, and the forwarding engine) -> 2 SSA ASICS (Super Santa Anna, fabric complex ASIC). The Metro Krypton ASIC (Metropolis Krypton) is an interface that sits between a port ASIC and the EARL complex and can handle Multicast. The Kuma ASIC handles the forwarding bus lookup.

For the WS-X6716-10G two ports share a single 10G port ASIC (the NAXOS ASIC) and two port pairs (two NAXOS ASICs) feed into a single ASHWINI ASIC which is also 10Gbps. These feed into 4x R2D2 then 2x Metro Argos then 2x Super Santa Ana's.

There is lots of info on 6500/7600 line card ASICs here: /index.php?page=6500-7600-qos#ports-asics-fabric

The 48x1G cards and 4x10G card have the same number of each ASIC (JANUS, SSA and ROHINI):

7600#show asic-version slot 1
Module in slot 1 has 3 type(s) of ASICs ! WS-X6748-GE-TX
        ASIC Name      Count      Version
            JANUS          2      (1.0)
              SSA          2      (9.0)
           ROHINI          4      (1.5)
7600#show asic-version slot 2
Module in slot 2 has 3 type(s) of ASICs ! WS-X6748-SFP
        ASIC Name      Count      Version
            JANUS          2      (1.0)
              SSA          2      (9.0)
           ROHINI          4      (1.6)
7600#show asic-version slot 3
Module in slot 3 has 3 type(s) of ASICs ! WS-X6704-10GE
        ASIC Name      Count      Version
            JANUS          2      (1.0)
              SSA          2      (9.0)
           ROHINI          4      (1.6)
7600#show asic-version slot 5
Module in slot 5 has 8 type(s) of ASICs ! WS-X6708-10GE
        ASIC Name      Count      Version
             KUMA          1      (3.0)
      METRO_ARGOS          1      (3.0)
    METRO_KRYPTON          1      (3.0)
              SSA          2      (9.0)
       SANTA_CRUZ          1      (3.0)
          TELESTO          1      (7.0)
          KAILASH          1      (2.4)
             R2D2          2      (3.0)

 

#Outputs from WS-X6708-10G

show asic slot  
ASIC Name Count Version
KUMA 2 (3.0)
METRO_ARGOS 2 (3.0)
METRO_KRYPTON 2 (3.0)
SSA 2 (9.0)
R2D2 8 (2.0)
TIANGANG 4 (54.0) 

Te/1
Ports-in-ASIC (Sub-port ASIC) : 1,4-5,7 (1)
Te/2
Ports-in-ASIC (Sub-port ASIC) : 2-3,6,8 (2) 


# Outputs from WS-X6716-10G

#show asic slot 1

Module in slot 1 has 7 type(s) of ASICs
ASIC Name Count Version
KUMA 2 (3.0)
METRO_ARGOS 2 (3.0)
METRO_KRYPTON 2 (3.0)
SSA 2 (9.0)
NAXOS 8 (1.0)
R2D2 4 (3.0)
ASHWINI 4 (0.6)


#show interface Te1/1 capabilities | inc ASIC
Ports-in-ASIC (Sub-port ASIC) : 1-8 (1-4)
#show interface Te1/5 capabilities | inc ASIC
Ports-in-ASIC (Sub-port ASIC) : 1-8 (5-8)
#show interface Te1/9 capabilities | inc ASIC
Ports-in-ASIC (Sub-port ASIC) : 9-16 (9-12)
#show interface Te1/13 capabilities | inc ASIC
Ports-in-ASIC (Sub-port ASIC) : 9-16 (13-16) 

For the WS-X6908-10G each port has a Ganita ASIC which is the PHY ASIC, each GANITA feeds into one R2D2 port ASIC (10G ASIC). The ports are paired 1&2, 3&4 etc, and feed into port-group ASICs (Metro Argos). Metro Argos > SSA (Super Santa Anna) ASICs provide the fabric (serial) connection > feeds into the Santa Monica ASIC providing the backplane connectivity.

! WS-X6908-10G
6509E-SUPT2T#show asic-version switch 1 slot 1
Module in switch 1, slot 1 has 7 type(s) of ASICs ! DCEF-2T WS-X6908-10G
        ASIC Name      Count      Version
      METRO_ARGOS          4      (4.0)
    METRO_KRYPTON          4      (4.0)
           MALFRA          2      (16.4)
              SSA          4      (9.0)
     SANTA_MONICA          1      (1.0)
           GANITA          8      (2.0)
             R2D2          8      (5.0)

Ports-in-ASIC (Sub-port ASIC) : 1-2 (1) Ports-in-ASIC (Sub-port ASIC) : 3-4 (3) Ports-in-ASIC (Sub-port ASIC) : 5-6 (5) Ports-in-ASIC (Sub-port ASIC) : 7-8 (7)

7600#show asic-version slot 1
Module in slot 1 has 5 type(s) of ASICs ! 7600-ES20-10G3CXL
ASIC Name Count Version
SSA 1 (0.0)
KRYPTON 1 (14.0)
BLACKWATER 1 (3.0)
XCHIP 1 (2.4)
RASP16 1 (2.2)

7600-sp#show platform hardware asic-versions
ASIC Name Version
slot : 1
ST_TYCHO (68) (3.2)
ST_SUPERMAN (67) (2.2)
BLACKWATER (62) (3.0)
XCHIP (63) (2.4)
RASP16 (64) (2.2)
KRYPTON (52) (14.0)
KUMA (70) (3.0)
SSA (43) (0.0)

 

Below are the card and ASIC layouts for the ES20 and ES40 cards:

From the various supervisor line cards the SANTA_CRUZ and SANTA_MONICA ASICs are some how related to the forwarding (they provide the backplane connectivity). The Trident line cards in the ASR9000 series are based on the NP3C network processing unit used in ES+ line cards (so they are basically beefed up ES+ line cards) and use the SANTA_CRUZ ASIC. The Santa Cruz ASIC was replaced by the Santa Monica ASIC. The Santa Cruz was 20 ports, 20Gbps max each == 360Gbps total. The Santa Monica suports 26 ports, 40Gbps max each == 1040Gbps.

ES20 line cards used the PXF (Parallel eXpress Forwarding) engine codenamed "Toaster".

On the SUP720 and RSP720 (with EARL7) the Superman ASIC performs the layer 2 forwarding lookup operations and the Tycho ASIC performs the layer 3 forwarding lookup operations. On the RSP720-3C/DFC3C EARL 7.5 is used, Superman and Tycho are one single ASIC called Supertycho so when performing an ELAM against the Superman ASIC Supertycho is presumably called as this seems to capture layer 2, MPLS and native layer 3 traffic. The link between Superman/Tycho/Supertycho and the SANTA_CRUZ / SANTA_MONICA ASICs is not clear. Also the KAILASH ASIC is actually an FPGA which has something to do with parsing RP and SP CPU punted/originated traffic. The TELESTO ASIC is something to do with fabric connectivity?

For SUP2T the Eureka and Lamira ASICs form the EARL8 chip set and the layer 2 forwarding engine, Eureka is used for ELAM captures on SUP2T/PFC4/DFC4. Below are outputs from a few 7600s and 6500s showing each supervisor has one of either the SANTA_MONICA or SANTA_CRUZ ASIC.

!
! 6509-E VSS stack with VS-SUP2T-10G
!

6509-E#show module switch 1 slot 5
 Switch Number:     1   Role:   Virtual Switch Active
----------------------  -----------------------------
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  5    5  Supervisor Engine 2T 10GE w/ CTS (Acti VS-SUP2T-10G       XXXXXXXXXXX

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  5  0006.f614.cc72 to 0006.f614.cc79   1.5   12.2(50r)SYS 15.1(1)SY1   Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  5  Policy Feature Card 4       VS-F6K-PFC4        XXXXXXXXXXX  2.1    Ok
  5  CPU Daughterboard           VS-F6K-MSFC5       XXXXXXXXXXX  2.1    Ok

Mod  Online Diag Status
---- -------------------
  5  Pass

6509-E#show asic-version switch 1 slot 5
Module in switch 1, slot 5 has 10 type(s) of ASICs
        ASIC Name      Count      Version
             KUMA          1      (3.0)
      METRO_ARGOS          1      (4.0)
    METRO_KRYPTON          1      (4.0)
              SSA          2      (9.0)
     SANTA_MONICA          1      (1.0)
          TELESTO          1      (8.0)
          KAILASH          1      (4.0)
           GANITA          2      (2.0)
             R2D2          2      (5.0)
          ALKINDI          1      (4.127)


!
! 7606-S with RSP720-3CXL-10G
!

7606-S#show module 5
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  5    5  Route Switch Processor 720 10GE (Activ RSP720-3CXL-10GE   XXXXXXXXXXX

Mod MAC addresses                       Hw   Fw            Sw           Status
--- ---------------------------------- ----- ------------- ------------ -------
  5  001c.584e.e8e4 to 001c.584e.e8eb  2.2   12.2(33r)SRC  15.3(3)S     Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  5  Policy Feature Card 3       7600-PFC3CXL-10GE  XXXXXXXXXXX  1.2    Ok
  5  C7600 MSFC4 Daughterboard   7600-MSFC4         XXXXXXXXXXX  2.1    Ok

Mod  Online Diag Status
---- -------------------
  5  Pass

7606-S#show asic-version slot 5
Module in slot 5 has 8 type(s) of ASICs
        ASIC Name      Count      Version
             KUMA          1      (3.0)
      METRO_ARGOS          1      (3.0)
    METRO_KRYPTON          1      (3.0)
              SSA          2      (9.0)
       SANTA_CRUZ          1      (3.0)
          TELESTO          1      (7.0)
          KAILASH          1      (2.4)
             R2D2          2      (3.0)


!
! 6513 (non-E) with SUP720
!

6513#show module 7
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  7    2  Supervisor Engine 720 (Active)         WS-SUP720-3B       XXXXXXXXXXX

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  7  0016.4708.3314 to 0016.4708.3317   5.3   8.4(2)       12.2(33)SXI9 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  7  Policy Feature Card 3       WS-F6K-PFC3B       XXXXXXXXXXX  2.3    Ok
  7  MSFC3 Daughterboard         WS-SUP720          XXXXXXXXXXX  2.6    Ok

Mod  Online Diag Status
---- -------------------
  7  Pass

6513#show asic-version slot 7
Module in slot 7 has 4 type(s) of ASICs
        ASIC Name      Count      Version
              SSA          1      (9.0)
              SSO          4      (2.0)
         HYPERION          1      (6.0)
         PINNACLE          1      (4.0)