Date created: Thursday, April 28, 2022 2:44:41 PM. Last modified: Thursday, April 28, 2022 4:47:53 PM
ASR9000 Fabric Capacity
References:
BRKARC-2003 - Cisco ASR 9000 System Architecture (? 2015)
BRKARC-2003 - Cisco ASR 9000 System Architecture (Barcelona 2019)
BRKARC-2003 - Cisco ASR 9000 System Architecture (Barcelona 2018)
BRKARC-2003 - Cisco ASR 9000 System Architecture (Las Vegas 2017)
Overview
On Cisco ASR9K series routers the line cards have a range of fabric connection counts and speeds. These fabric connections connect a line card to either a dedicated fabric card, or fabric capacity on an RSP card, or both. The number and speed of the fabric connections on ASR9K lines cards varies by line card generation (Trident, Typhoon, Tomahawk, Lightspeed), even cards of the same generation can have different numbers of fabric connections, and two cards of mixed generations in the same chassis will operate in different modes depending on the fabric card and/or RSP card generation.
Note: Each fabric connection from a line card to a fabric card or RSP card is two fabric "channels". E.g. each of the four fabric connections on a Typhoon line card is 2x55Gbps fabric channels, meaning that one fabric connection is 110Gbps. Also note that when traffic is transported over the chassis fabric there is overhead (e.g. superframing) which means that in the example of a Typhoon line card, 110Gbps is not the available capacity across the fabric for traffic transiting the router.
Cards
Line Cards:
- Typhoon cards support up to 4 fabric connections to each fabric card / RSP. Each fabric connection on a Typhoon cards is 110Gbps (2x55Gbps fabric channels). 4 fabric connection Typhoon cards support 4x2x55Gbps == 440Gbps fabric capacity.
- A 5 fabric channel Tomahawk line card supports up to 5 connections to each fabric card / RSP. A 7 fabric channel Tomahawk line cards supports up to 7 connections to each fabric card / RSP. A fabric connection on a Tomahawk line card is 230Gbps (2x115Gbps fabric channels). 5 fabric connection Tomahawk cards support 5x2x115Gbps == 1,150Gbps of fabric capacity. 7 fabric connection cards support 7x2x115Gbps == 1,610Gbps of fabric capacity.
- Lightspeed cards support up to 7 fabric connections to each fabric card / RSP. Each fabric connection on a Lightspeed card is 600Gbps (2x300Gbps fabric channels), 7x2x300Gbps == 4.2Tbps total fabric capacity.
Switch Fabric Cards:
Each SFC provides one fabric connection (which is two fabric channels) to each line card slot. The fabric bandwidth provided per-slot by each generate SFC is as follows:
– A single SFC v1 (Typhoon generation): 110G capacity to each line card (2x55Gbps channels)
– A single SFC v2 (Tomahawk generation): 230G capacity to each line card (2x115Gbps channels)
– A single SFC v3 (Lightspeed generation): 600G capacity to each line card (2x300Gbps channels)
RSP Cards:
When using RSP cards as a fabric card in a chassis where there are no fabric cards (ASR9904/9006/9010), the RSPs provide all the fabric connections which the line cards can support (multiple connections to each card). When mixed with fabric cards in chassis that support SFCs, the RSPs only provide one fabric channel to each line card slot not two. like fabric cards. The fabric bandwidth provided per-slot by each RSP type is as follows:
- A single RSP440 (any chassis): Always provides 220Gbps capacity to each line card (for all line cards)
- A single RSP880 (ASR9006/9010 chassis): Can support upto 460Gbps capacity to each line card (only 4 fabric channels per line card is supported in this chassis, 4x115Gbps channels provided).
- A single RSP880 (ASR9904 chassis): Can support upto 575Gbps capacity to each line card (for 5 fabric channel line cards, 5x115Gbps channels provided).
- A single RSP880 (ASR9904 chassis): Can support upto 805Gbps capacity per slot (for 7 fabric channel line cards, 7x115Gbps channels provided).
Examples by Fabric Card
Single RSP880 in ASR9006/9010 provides 2 fabric connections per line card slot; 2x2x115Gbps == 460Gbps/slot for Tomahawk line cards, 2x2x55Gbps == 220Gbps for Typhoon line cards.
Dual RSP880s in ASR9006/9010 provides 4 fabric connections per line card slot; 4x2x115Gbps == 920Gbps/slot for Tomahawk line cards, 4x2x55Gbps == 440Gbps for Typhoon line cards.
SFC v2 (Tomahawk generation):
5x SFC v2 cards (e.g. ASR9906) provides 5x230Gbps fabric connections (or 5x2x115Gbps fabric channels) == 1.15Tbps/slot for 5 fabric-connection Tomahawk line cards.
7x SFC v2 cards (e.g. ASR9912) provides 7x230Gbps fabric connections (or 7x2x115Gbps fabric channels) == 1.61Tbps/slot for 7 fabric-connection Tomahawk line cards.
7x SFC v2 cards in a 6+1 redundancy configuration provides 6x230Gbps fabric connections (or 6x2x115Gbps fabric channels) == 1.38Tbps/slot for 7 fabric connection Tomahawk line cards.
SFC v3 (Lightspeed generation):
5x600Gbps == 3.0Tbps/slot
5x SFC v3 cards (e.g. ASR9912) provides 5x600Gbps fabric connections (or 5x2x300Gbps) == 3.0Tbps/slot for 5 fabric-connections Lightspeed line cards.
7x SFC v3 cards (e.g. ASR9906/9910/9912/9922) provides 7x600Gbps fabric connections (or 7x2x300Gbps fabric channels) == 4.2Tbps/slot for 7 fabric connection Lightspeed line cards.
7x SFC v3 cards in a 6+1 redundancy configuration provides 6x600Gbps fabric connections (or 6x2x300Gbps fabric channels) == 3.6Tbps/slot for 7 fabric connection Lightspeed line cards.
Examples by Line Card
Typhoon Cards in ASR9006/9010 with only RSP440s providing fabric connectivity:
- Each RSP440 card provides 2x110Gbps fabric connections (2x2x55Gbps fabric channels) == 220Gbps/per line card slot of full-duplex fabric capacity
- Dual RSP440 cards provide 4x110Gbps fabric connections (4x2x55Gbps fabric channels) == 440Gbps of non-resilient fabric capacity or 220Gbps of non-resilient fabric capcity per line card slot
- Each Typhoon card supports 2x110Gbps fabric connections (2x2x55Gbps fabric channels) == 220Gbps/per line card of full-duplex fabric capacity with a single RSP440
- Each Typhoon card supports 4x110Gbps fabric connections (4x2x55Gbps fabric channels) == 440Gbps/per line card of full-duplex fabric capacity with dual RSP440s
Typhoon Cards in ASR9006/9010 with only RSP880s providing fabric connectivity:
- Each RSP880 supports 460Gbps of full-duplex fabric capacity (4x115Gbps fabric channels) per line card slot in ASR9006/ASR9010
- Each Tomahawk card supports 2x110Gbps fabric connections (2x2x55Gbps fabric channels) == 220Gbps/per slot full-duplex non-resilient with a single RSP880
- Each Tomahawk card supports 2x4x55Gbps fabric channels == 220Gbps/per slot full-duplex resilient with dual RSP880s
- Each Tomahawk card supports 2x4x55Gbps fabric channels == 440Gbps/per slot full-duplex non-resilient with dual RSP880s
Tomahawk Cards in ASR9006/9010 with only RSP880s providing fabric connectivity:
- Each RSP880 supports 460Gbps full-duplex fabric capacity (4x115Gbps fabric channels) per line card slot
- Each Tomahawk card supports 4x115Gbps fabric channels == 460Gbps/per slot full-duplex non-resilient with a single RSP880
- Each Tomahawk card supports 2x4x115Gbps fabric channels == 460Gbps/per slot full-duplex resilient with dual RSP880s
- Each Tomahawk card supports 2x4x115Gbps fabric channels == 920Gbps/per slot full-duplex non-resilient with dual RSP880s
Tomahawk Cards in ASR9904 with only RSP880s providing fabric connectivity:
- Each RSP880 supports 575Gbps full-duplex fabric capacity (5x115Gbps fabric channels) per line card slot in an ASR9904
- A 5-fabric channel A9K-XX Tomahawk card supports 5x115Gbps fabric channels == 575Gbps/per slot full-duplex fabric capacity
- A 5-fabric channel A9K-XX Tomahawk card supports 2x5x115Gbps fabric channels == 575Gbps/per slot full-duplex resilient with dual RSP880s
- A 5-fabric channel A9K-XX Tomahawk card supports 2x5x115Gbps fabric channels == 1.15Tbps/per slot full-duplex non-resilient with dual RSP880s
- A 7-fabric channel A99-XX Tomahawk card supports 7x115Gbps fabric channels == 805Gbps/per slot full-duplex non-resilient with a single RSP880
- A 7-fabric channel A99-XX Tomahawk card supports 2x7x115Gbps fabric channels == 805Gbps/per slot full-duplex resilient with dual RSP880s
- A 7-fabric channel A99-XX Tomahawk card supports 2x7x115Gbps fabric channels == 1.61Tbps/per slot full-duplex non-resilient with dual RSP880s
A9K-8X100G Tomahawk Line Card:
- Supports 5 fabric cards to each fabric / RSP card
- 10x115Gbps fabric links in total
- 5x2x115Gbps (575Gbps) to each fabric card
A99-8X100GE:
- Supports 7 fabric cards
- 7x 215Gbps channels
A99-12X100G:
- Supports 7 fabric cards
- 14x115Gbps fabric links
- 2x115Gbps to each fabric card
- NPU <> FIA: Interlaken = 2x 16x 10.9375Gbps = 350G raw BW
- FIA <> SM15: 2x 8x15G = 240Gbps raw BW, 3x Fencer ports
16X100G Lightspeed Line Card:
- 4 slices with 4x 100G ports each
- 5 Fabric channel line card
In ASR9906/9910/9912/9922:
- 43.75G x 2x8 lanes x 5 fab
= 3.5T (raw)
= 3.0T (actual)
In ASR9006/9010 (only 4 fabric channels can be used and slower speed):
- 33.75G x 4x8 lanes x 2RSP
= 2.16T (raw)
= 1.8T (actual)
= 900G per RSP
A99-32x100G-TR 32X100G Lightspeed Line Card:
- 8 slices with 4x100G ports each
- 7 Fabric channel line card
- 2x SKB Switch Fabrics, each has 7x300Gbps full duplex fabric channels (2.1Tbps per SKB) for a total of 4.2Tbps full duplex fabric capacity per line card
- Requires light speed fabric (RP3 / SFC3 / RSP5) in ASR9922 / ASR9912 / ASR9910 / ASR9906 or ASR9904 chassis
- 3.2Tbps line rate in all chassis with full redundancy except ASR9904 with single RSP5
Typhoon NPU TM supports ~70Gbps
Tomahawk NPU TM supports 200 or 240Gbps?
Lightspeed?
Examples by Chassis
Single RSRP880 in ASR9904 provides 5x115Gbps == 575Gbps/slot for 5 fabric channel line cards like the A9K-8X100G-TR.
Dual RSP880s in ASR9904 provides 2x5x115Gbps == 1.15Tbps/slot for 5 fabric channel line cards like the A9K-8X100G-TR.
Single RSP880 in ASR9904 provides 7x115Gbps == 805Gbps/slot for 7 fabric channel line cards like the A99-8X100G-TR.
Dual RSP880s in ASR9904 provides 2x7x115Gbps == 1.61Tbps/slot for 5 fabric channel line cards like the A99-8X100G-TR.
Single RSP5 in ASR9006/9010 provides ???
Single RSP5 in ASR9904 provides 1.8Tbps to Lightspeed cards
Dual RSP5s in ASR9904 provides 3.2Tbps to Lightspeed cards
Single RSP5 in ASR9906/9910/9912/9922 provides 7x600Gbps == 4.2Tbps/slot for 7 fabric channel line cards like the A99-8X100G-TR.
Single RSP5 in ASR9910 with 24x10G cards got 230Gbps throughput, because RSP5 provides 230Gbps SF capacity?
Lightspeed Cards in ASR9906/9910/9912/9922 with RSP5/SFC v3 for fabric:
- Each RSP5 supports 600Gx7Fab = 4.2T/slot
- 3.2T linerate with fabric redundancy
Lightspeed Cards in ASR9904:
- 3.2T linerate with dual RSP5
- 1.8T throughput with single RSP5
Under Capacity Example
This example ASR9904 chassis has two A9K-8X100GE-TR Tomahawk generation line cards and one A9K-RSP5-TR Lightspeed generation RSP. It has logged the follow message on boot:
LC/0/1/CPU0:Apr 10 13:51:34.713 UTC: pfm_node_lc[326]: %FABRIC-FIA-1-RATE_LIMITER_ON : Set|fialc[4522]|0x108a000|Insufficient fabric capacity for card types in use - FIA egress rate limiter applied
RP/0/RSP0/CPU0:iosxr#show pfm location all
node: node0_0_CPU0
---------------------
CURRENT TIME: Apr 11 09:13:15 2019
PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Apr 8 08:51:27 2019|0 |RATE_LIMITER_ON |E/A|4509 |Fabric Interfa|0x108a000
node: node0_1_CPU0
---------------------
CURRENT TIME: Apr 11 09:13:15 2019
PFM TOTAL: 0 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Mon Jan 01 00:00:000|--|NONE |NO |0000000|NONE |0x00000000
node: node0_RSP0_CPU0
---------------------
CURRENT TIME: Apr 11 09:13:15 2019
PFM TOTAL: 0 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Mon Jan 01 00:00:000|--|NONE |NO |0000000|NONE |0x00000000
Below is output from the same chassis. The from the first line card shown has 2 slices (4 ports) powered down, 2 slices (4 ports) are powered up. The output from the second line card shown has all 4 slices (8 ports) powered up:
RP/0/RSP0/CPU0:iosxr#show run | inc hw-mod
Building configuration...
hw-module location 0/1/CPU0 slice 2 power-savings
hw-module location 0/1/CPU0 slice 3 power-savings
RP/0/RSP0/CPU0:iosxr#show controllers fabric fia information location 0/1/CPU0
********** FIA-0 **********
Category: bandwidth-0
BW if0 120
BW if1 120
********** FIA-1 **********
Category: bandwidth-1
BW if0 120
BW if1 120
********** FIA-2 **********
Category: bandwidth-2
BW if0 89
BW if1 89
********** FIA-3 **********
Category: bandwidth-3
BW if0 67
BW if1 67
RP/0/RSP0/CPU0:iosxr#show controllers fabric fia information location 0/0/CPU0
********** FIA-0 **********
Category: bandwidth-0
BW if0 67
BW if1 67
********** FIA-1 **********
Category: bandwidth-1
BW if0 67
BW if1 67
********** FIA-2 **********
Category: bandwidth-2
BW if0 67
BW if1 67
********** FIA-3 **********
Category: bandwidth-3
BW if0 67
BW if1 67
After inserting a 2nd RSP5 the error is cleared and all cards/ports support line rate:
RP/0/RSP0/CPU0:iosxr#show controllers fabric fia information location 0/0/CPU0
********** FIA-0 ********** Category: bandwidth-0 BW if0 120 BW if1 120 ********** FIA-1 ********** Category: bandwidth-1 BW if0 120 BW if1 120 ********** FIA-2 ********** Category: bandwidth-2 BW if0 120 BW if1 120 ********** FIA-3 ********** Category: bandwidth-3 BW if0 120 BW if1 120
RP/0/RSP0/CPU0:iosxr#show controllers fabric fia information location 0/1/CPU0
********** FIA-0 ********** Category: bandwidth-0 BW if0 120 BW if1 120 ********** FIA-1 ********** Category: bandwidth-1 BW if0 120 BW if1 120 ********** FIA-2 ********** Category: bandwidth-2 BW if0 120 BW if1 120 ********** FIA-3 ********** Category: bandwidth-3 BW if0 120 BW if1 120
Previous page: ASR9000 ASR9K-VSM-500
Next page: ASR9000 NP Packet Counter Capture