Date created: Tuesday, December 15, 2015 2:55:01 PM. Last modified: Thursday, April 28, 2022 2:45:00 PM
ASR9001 Specific Hardware Overview
References:
http://www.cisco.com/c/en/us/products/collateral/routers/asr-9001-router/data_sheet_c78-685687.html
http://www.cisco.com/c/en/us/support/docs/routers/asr-9000-series-aggregation-services-routers/200135-Aggregation-Services-Router-9001-ASR-90.html#anc13
BRKARC-2003 - Cisco ASR9000 Architecture and Network Virtualisation
BRKSPG-2904 - ASR-9000 IOS-XR Hardware Architecture, QOS, EVC, IOS-XR Configuration and Troubleshooting
http://www.cisco.com/c/en/us/support/docs/routers/asr-9000-series-aggregation-services-routers/116999-problem-line-card-00.html
The ASR9001 "Iron Man" has 1 built in RSP with quad-core PPC CPU and 8GBs of RAM. The chassis supports 120Gbps of non-blocking full-duplex traffic (dual Typhoon chips). It has 4x 10Gbps SFP+s ports built in and 2x Modular Port Adapters (MPAs) bays that will support the 20x 1Gbps / 2x 10Gbps / 4x 10Gbps / 1x 40Gbps line cards. The internals of the ASR9001 (the NPUs and the built in 4x10G SFP+ ports) are built using the 2nd generation Typhoon line card hardware (which support four million IPv4 or two million IPv6 prefixes in shared RLDRAM by default, without changing the scaling profile, and one million MPLS labels).
Even though the ASR9001 supports OIR, all ports on the NP attached to the bay the new MPA is being inserted into will experience a brief outage (should be sub-second) as all ports go through a fast reset.
The ASR9001-S is the exact same unit as the ASR9001 but two of the built in 10G SFP+ ports and the 2nd MPA slot are disabled, but can be enabled with a license upgrade and reboot.
The 9001 chassis has two Network Processors Units (NPUs/NPs), so for the ASR9001-S the 2nd NP is disabled. The first MPA bay and 2 of the built in SFP+ ports are on one NP and the 2nd bay and remaining two built in SFP+ ports are on the other NP. These are Typhoon NPs that have 6x 10G lanes, so 60Gbps of non-blocking throughput per NPU, hense 120Gbps non-blocking per chassis.
The example ASR9001 chassis below has a 20x1G MPA in bay 0 (Gi0/0/0/0 to Gi0/0/0/19) and a 4x 10G MPA in bay 1 (Te0/0/1/0 to Te0/0/1/3). The built in SFP+ ports (Te0/0/2/0 to Te0/0/2/4) sit in "virtual" bay 2. Altogether, bay 0 (0/0/0/x) and bay 1 (0/0/1/x) sit on an internal Modular Line Card which shows as 0/0/CPU (like a MOD80-SE MPA). Bay 2 sits (0/0/2/x) sits on a Virtual Module and they all share the MPA same CPU, 0/0/CPU0. So the port numbers are Chassis/MPA/Bay/Port.
For comparison, in an ASR9006 for example with two A9K-MOD80-SE "80G Modular Linecard, Service Edge Optimized" MPAs, and a single 4x10G line card in one bay of each of the MOD80-SE MPAs, the first card shows as Te0/0/0/0 to Te0/0/0/3 and the second as Te0/1/0/0 to Te0/1/0/3:
RP/0/RSP0/CPU0:abr1#show inventory
Tue Dec 15 15:02:27.735 UTC
NAME: "module 0/RSP0/CPU0", DESCR: "ASR 9001, Route Switch Processor with 8GB memory"
PID: ASR9001-RP, VID: V01 , SN: xxxxx
NAME: "module 0/0/CPU0", DESCR: "ASR 9001, Modular Line Card"
PID: ASR9001-LC, VID: V01 , SN: xxxxxx
NAME: "module 0/0/0", DESCR: "ASR 9000 20-port 1GE Modular Port Adapter"
PID: A9K-MPA-20X1GE, VID: V02, SN: xxxxxx
NAME: "module 0/0/1", DESCR: "ASR 9000 4-port 10GE Modular Port Adapter"
PID: A9K-MPA-4X10GE, VID: V05, SN: xxxxx
NAME: "module mau 0/0/1/0", DESCR: "Multirate 10GBASE-LR and OC-192/STM-64 SR-1 XFP, SMF"
PID: XFP-10GLR-OC192SR , VID: V04 , SN: xxxxx
NAME: "module 0/0/2", DESCR: "ASR 9000 Virtual Module"
PID: A9K-MODULEv, VID: N/A, SN: N/A
NAME: "module mau 0/0/2/0", DESCR: "10GBASE-LR SFP+ Module for SMF"
PID: SFP-10G-LR , VID: V01 , SN: xxxx
NAME: "module mau 0/0/2/1", DESCR: "10GBASE-LR SFP+ Module for SMF"
PID: SFP-10G-LR , VID: V01 , SN: xxxxx
NAME: "module mau 0/0/2/2", DESCR: "10GBASE-LR SFP+ Module for SMF"
PID: SFP-10G-LR , VID: V02 , SN: xxxxx
NAME: "module mau 0/0/2/3", DESCR: "10GBASE-LR SFP+ Module for SMF"
PID: SFP-10G-LR , VID: V01 , SN: xxxxxx
RP/0/RSP0/CPU0:abr1#show hw-module fpd location all
Tue Dec 15 14:03:27.705 UTC
===================================== ==========================================
Existing Field Programmable Devices
==========================================
HW Current SW Upg/
Location Card Type Version Type Subtype Inst Version Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/RSP0/CPU0 ASR9001-RP 1.0 lc fpga2 0 1.14 No
lc cbc 0 22.114 No
lc rommon 0 2.01 No
--------------------------------------------------------------------------------
0/FT0/SP ASR-9001-FAN 1.0 ft cbc 7 24.114 No
--------------------------------------------------------------------------------
0/0/CPU0 ASR9001-LC 1.0 lc cbc 0 23.114 No
lc fpga4 0 2.07 No
lc fpga2 0 1.18 No
lc rommon 0 2.01 No
--------------------------------------------------------------------------------
0/0/0 A9K-MPA-20X1GE 1.0 spa fpga3 0 0.08 No
--------------------------------------------------------------------------------
0/0/1 A9K-MPA-4X10GE 1.0 spa fpga6 1 1.06 No
--------------------------------------------------------------------------------
RP/0/RSP0/CPU0:abr1#show platform summary location all
Tue Dec 15 14:06:13.257 UTC
-------------------------------------------------------------------------------
Platform Node : 0/RSP0/CPU0 (slot 0)
PID : ASR9001-RP
Card Type : ASR 9001, Route Switch Processor with 8GB memory
VID/SN : V01 / xxxxxx
Oper State : IOS XR RUN
Last Reset : User Initiated reload Process: reload
: Tue Jan 27 00:27:55 2015
Configuration : Power is enabled
Bootup enabled.
Monitoring enabled
Rommon Ver : Version 2.01(20130627:172232)
IOS SW Ver : 4.3.4
Main Power : Power state Enabled. Estimate power 50 Watts of power required.
Faults : N/A
-------------------------------------------------------------------------------
Platform Node : 0/0/CPU0 (slot 2)
PID : ASR9001-LC
Card Type : ASR 9001, Modular Line Card
VID/SN : V01 / xxxxx
Oper State : IOS XR RUN
Last Reset : MBI-HELLO powering down node on receiving powerdown notification Process: mbi-hello
: Tue Mar 4 12:31:14 2014
Configuration : Power is enabled
Bootup enabled.
Monitoring enabled
Rommon Ver : Version 2.01(20130628:060031)
IOS SW Ver : 4.3.4
Main Power : Power state Enabled. Estimate power 330 Watts of power required.
Faults : N/A
-------------------------------------------------------------------------------
Platform Node : 0/0/0 (slot 2, bay 0)
PID : A9K-MPA-20X1GE
Card Type : ASR 9000 20-port 1GE Modular Port Adapter
VID/SN : V02 / xxxxxx
Oper State : OK
Last Reset : Tue Jan 27 00:35:07 2015
Up Time : 46w0d
Configuration : Power is enabled
Bootup enabled
Monitoring enabled
FPD SW Rev(s) :
-------------------------------------------------------------------------------
Platform Node : 0/0/1 (slot 2, bay 1)
PID : A9K-MPA-4X10GE
Card Type : ASR 9000 4-port 10GE Modular Port Adapter
VID/SN : V05 / xxxxxx
Oper State : OK
Last Reset : Tue Jan 27 00:34:49 2015
Up Time : 46w0d
Configuration : Power is enabled
Bootup enabled
Monitoring enabled
FPD SW Rev(s) :
-------------------------------------------------------------------------------
The ASR9001 has two NPs. The output below shows which ports are mapped to which NP and which Fabric Interconnect ASIC:
RP/0/RSP0/CPU0:abr1#show controllers np summary all
Tue Dec 15 14:20:53.225 UTC
Node: 0/0/CPU0:
----------------------------------------------------------------
[total 2 NP] Driver Version 12.46a Build 3
NP 0 : Hardware Revision: v2 A0
: Ucode - Version: 1.0 Build Date: ( Nov 9 2015, 20:50:00 )
NP 1 : Hardware Revision: v2 A0
: Ucode - Version: 1.0 Build Date: ( Nov 9 2015, 20:50:00 )
RP/0/RSP0/CPU0:abr1#show controllers np ports all
Tue Dec 15 14:21:52.537 UTC
Node: 0/0/CPU0:
----------------------------------------------------------------
NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 0 0 GigabitEthernet0/0/0/0 - GigabitEthernet0/0/0/19
0 0 0 TenGigE0/0/2/0, TenGigE0/0/2/1
1 1 1 TenGigE0/0/1/0, TenGigE0/0/1/1, TenGigE0/0/1/2, TenGigE0/0/1/3
1 1 1 TenGigE0/0/2/2, TenGigE0/0/2/3
RP/0/RSP0/CPU0:abr1#show controllers np portMap all
Tue Dec 15 14:31:51.914 UTC
Node: 0/0/CPU0:
----------------------------------------------------------------
Show Port Map for NP: 0, and RX Unicast Ports
phy port num interface desc uiMappedSourcePort
0 UNDEFINED
1 UNDEFINED
2 UNDEFINED
3 UNDEFINED
4 UNDEFINED
5 UNDEFINED
6 UNDEFINED
7 UNDEFINED
8 TenGigE0/0/2/0 63 (unmapped)
9 TenGigE0/0/2/1 63 (unmapped)
10 UNDEFINED
11 Egress Fabric1 63 (unmapped)
12 Egress Fabric2 63 (unmapped)
13 UNDEFINED
14 UNDEFINED
15 UNDEFINED
16 GigE0/0/0/0 63 (unmapped)
17 GigE0/0/0/1 63 (unmapped)
18 GigE0/0/0/2 63 (unmapped)
19 GigE0/0/0/3 63 (unmapped)
20 GigE0/0/0/8 63 (unmapped)
21 GigE0/0/0/9 63 (unmapped)
22 Unknown 63 (unmapped)
23 Unknown 63 (unmapped)
24 GigE0/0/0/10 63 (unmapped)
25 GigE0/0/0/11 63 (unmapped)
26 GigE0/0/0/12 63 (unmapped)
27 GigE0/0/0/13 63 (unmapped)
28 GigE0/0/0/18 63 (unmapped)
29 GigE0/0/0/19 63 (unmapped)
30 Unknown 63 (unmapped)
31 Unknown 63 (unmapped)
32 UNDEFINED
33 UNDEFINED
34 UNDEFINED
35 UNDEFINED
36 UNDEFINED
37 UNDEFINED
38 UNDEFINED
39 UNDEFINED
40 GigE0/0/0/4 63 (unmapped)
41 GigE0/0/0/5 63 (unmapped)
42 GigE0/0/0/6 0 (bundle)
43 GigE0/0/0/7 0 (bundle)
44 UNDEFINED
45 UNDEFINED
46 UNDEFINED
47 UNDEFINED
48 GigE0/0/0/14 63 (unmapped)
49 GigE0/0/0/15 63 (unmapped)
50 GigE0/0/0/16 63 (unmapped)
51 GigE0/0/0/17 63 (unmapped)
52 UNDEFINED
53 UNDEFINED
54 UNDEFINED
55 UNDEFINED
56 UNDEFINED
57 UNDEFINED
58 UNDEFINED
59 UNDEFINED
60 UNDEFINED
61 UNDEFINED
62 UNDEFINED
63 UNDEFINED
Show Port Map for NP: 1, and RX Unicast Ports
phy port num interface desc uiMappedSourcePort
0 UNDEFINED
1 UNDEFINED
2 UNDEFINED
3 UNDEFINED
4 TenGigE0/0/1/0 63 (unmapped)
5 TenGigE0/0/1/1 63 (unmapped)
6 TenGigE0/0/1/2 63 (unmapped)
7 TenGigE0/0/1/3 63 (unmapped)
8 TenGigE0/0/2/2 63 (unmapped)
9 TenGigE0/0/2/3 63 (unmapped)
10 UNDEFINED
11 Egress Fabric1 63 (unmapped)
12 Egress Fabric2 63 (unmapped)
13 UNDEFINED
14 UNDEFINED
15 UNDEFINED
16 UNDEFINED
17 UNDEFINED
18 UNDEFINED
19 UNDEFINED
20 UNDEFINED
21 UNDEFINED
22 UNDEFINED
23 UNDEFINED
24 UNDEFINED
25 UNDEFINED
26 UNDEFINED
27 UNDEFINED
28 UNDEFINED
29 UNDEFINED
30 UNDEFINED
31 UNDEFINED
32 UNDEFINED
33 UNDEFINED
34 UNDEFINED
35 UNDEFINED
36 UNDEFINED
37 UNDEFINED
38 UNDEFINED
39 UNDEFINED
40 UNDEFINED
41 UNDEFINED
42 UNDEFINED
43 UNDEFINED
44 UNDEFINED
45 UNDEFINED
46 UNDEFINED
47 UNDEFINED
48 UNDEFINED
49 UNDEFINED
50 UNDEFINED
51 UNDEFINED
52 UNDEFINED
53 UNDEFINED
54 UNDEFINED
55 UNDEFINED
56 UNDEFINED
57 UNDEFINED
58 UNDEFINED
59 UNDEFINED
60 UNDEFINED
61 UNDEFINED
62 UNDEFINED
63 UNDEFINED
The ASR-9001 has a single Fabric Switching ASIC (Sacramento) at location 0/0/CPU0 and two Fabric Interface ASIC (FIAs). The Fabric Switching ASIC (Skytrain) is the same ASIC that is used on the RSP440 and 2nd generation based linecards on other ASR9000 platforms. The Fabric Switching ASIC has 4 ports with 2 connected to each FIA, each FIA connects to one of the two Trident NPUs and like the NPUs has 60Gbps of non-blocking throughput capacity.
The following diagram shows this toplogy:
The following command displays the counters related to the Fabric Switching ASIC:
RP/0/RSP0/CPU0:abr1#show controllers fabric crossbar statistics instance 0 location 0/0/CPU0
Tue Dec 15 15:13:54.152 UTC
Port statistics for xbar:0 port:1
==============================
Hi priority stats (unicast)
===========================
Ingress Packet Count Since Last Read : 1673157979
Egress Packet Count Since Last Read : 1026151308
Low priority stats (multicast)
===========================
Ingress Packet Count Since Last Read : 11895
Port statistics for xbar:0 port:2
==============================
Hi priority stats (unicast)
===========================
Ingress Packet Count Since Last Read : 1677497411
Egress Packet Count Since Last Read : 1026151307
Low priority stats (multicast)
===========================
Ingress Packet Count Since Last Read : 16856
Egress Packet Count Since Last Read : 2595832
Port statistics for xbar:0 port:3
==============================
Hi priority stats (unicast)
===========================
Ingress Packet Count Since Last Read : 6568112590
Ingress Channel Utilization Count : 2
Egress Packet Count Since Last Read : 7209854261
Egress Channel Utilization Count : 2
Low priority stats (multicast)
===========================
Ingress Packet Count Since Last Read : 2714700
Egress Packet Count Since Last Read : 2743723
Port statistics for xbar:0 port:4
==============================
Hi priority stats (unicast)
===========================
Ingress Packet Count Since Last Read : 6553243164
Ingress Channel Utilization Count : 2
Egress Packet Count Since Last Read : 7209854283
Egress Channel Utilization Count : 2
Low priority stats (multicast)
===========================
Ingress Packet Count Since Last Read : 148481
Egress Packet Count Since Last Read : 148208
Total Unicast In: 16472011144
Total Unicast Out: 16472011159
Total Multicast In: 2891932
Total Multicast Out: 5487763
RP/0/RSP0/CPU0:abr1#show controllers fabric crossbar link-status instance 0 location 0/0/CPU0
Tue Dec 15 15:20:21.033 UTC
PORT Remote Slot Remote Inst Logical ID Status
======================================================
01 02 01 1 Up
02 02 01 0 Up
03 02 00 0 Up
04 02 00 1 Up
The following commands shows the counters for the FIAs
RP/0/RSP0/CPU0:abr1#show controllers fabric fia link-status location 0/0/CPU0
Tue Dec 15 15:19:35.577 UTC
********** FIA-0 **********
Category: link-0
spaui link-0 Up
spaui link-1 Up
arb link-0 Up
xbar link-0 Up
xbar link-1 Up
********** FIA-1 **********
Category: link-1
spaui link-0 Up
spaui link-1 Up
arb link-0 Up
xbar link-0 Up
xbar link-1 Up
RP/0/RSP0/CPU0:abr1#show controllers fabric fia stats location 0/0/CPU0
Tue Dec 15 15:15:50.506 UTC
********** FIA-0 **********
Category: count-0
From Unicast Xbar[0] 7228789172
From Unicast Xbar[1] 7228789179
From Unicast Xbar[2] 0
From Unicast Xbar[3] 0
From MultiCast Xbar[0] 2754474
From MultiCast Xbar[1] 148488
From MultiCast Xbar[2] 0
From MultiCast Xbar[3] 0
To Unicast Xbar[0] 6585755334
To Unicast Xbar[1] 6570846462
To Unicast Xbar[2] 0
To Unicast Xbar[3] 0
To MultiCast Xbar[0] 2725357
To MultiCast Xbar[1] 148812
To MultiCast Xbar[2] 0
To MultiCast Xbar[3] 0
To Line Interface[0] 3607130880
To Line Interface[1] 10945907385
From Line Interface[0] 3693656345
From Line Interface[1] 9530309407
Ingress drop: 6
Egress drop: 0
Total drop: 6
********** FIA-1 **********
Category: count-1
From Unicast Xbar[0] 1029084883
From Unicast Xbar[1] 1029084885
From Unicast Xbar[2] 0
From Unicast Xbar[3] 0
From MultiCast Xbar[0] 2606325
From MultiCast Xbar[1] 0
From MultiCast Xbar[2] 0
From MultiCast Xbar[3] 0
To Unicast Xbar[0] 1681749349
To Unicast Xbar[1] 1677400227
To Unicast Xbar[2] 0
To Unicast Xbar[3] 0
To MultiCast Xbar[0] 16882
To MultiCast Xbar[1] 11912
To MultiCast Xbar[2] 0
To MultiCast Xbar[3] 0
To Line Interface[0] 1383470551
To Line Interface[1] 680999648
From Line Interface[0] 2792166675
From Line Interface[1] 601244228
Ingress drop: 7
Egress drop: 0
Total drop: 7
One can see errors for all ASICs with the following:
RP/0/RSP0/CPU0:abr1-lhclon.core#show asic-errors all location 0/0/CPU0 Tue Dec 15 16:08:31.297 UTC ************************************************************ * Fia ASIC Error Summary * ************************************************************ Instance : 0 Number of nodes : 0 SBE error count : 0 MBE error count : 0 Parity error count : 0 CRC error count : 0 Generic error count : 0 Reset error count : 0 -------------------- Instance : 1 Number of nodes : 0 SBE error count : 0 MBE error count : 0 Parity error count : 0 CRC error count : 0 Generic error count : 0 Reset error count : 0 -------------------- ************************************************************ * Prm_np ASIC Error Summary * ************************************************************ Instance : 0 Number of nodes : 0 SBE error count : 0 MBE error count : 0 Parity error count : 0 CRC error count : 0 Generic error count : 0 Reset error count : 0 -------------------- Instance : 1 Number of nodes : 0 SBE error count : 0 MBE error count : 0 Parity error count : 0 CRC error count : 0 Generic error count : 0 Reset error count : 0 --------------------
The hardware packet flow is as follows:
- For MPA cards: MPA Port PHYs > NP > FIA > FSA > to FIA on another MPA > NP > PHY
- For the built int SFP+ ports: SFP+ PHY > direct to NP > FIA > FSA > FIA > NP > PHY
- There is 40Gbps of bandwidth available to each bay from each NPU (20Gbps is used by the 2 built in SFP+ ports assigned to each NPU) so the chassis cannot be oversubscribed
- Each Line Card has a LC CPU accessed via the NP (PHY > NP > LC CPU) for features like programming hardware, inline NetFlow, SW switched packets
The NP contains the lookup table (FIB, MAC), TCAM, Stats memory, Frame memory and performs LPTS (CoPP). Each line card has a CPU that provides local logic to run control plane protocols like BFD, ARP, ICMP, OAM NetFlow etc all distributed at the line-card level.
Larger routers and line cards have a bridges that sits between the NPs and FIAs, it sits as a non-blocking memory converter between the two and can be checked using "show controllers fabric fia bridge stats location 0/RSP0/CPU0". On the 9001 there is one bridge per NP and they cannot be interacted with.
Down at the interface level, the four built-in SFP+ interface are 10G only, not mixed 1/10G. If fragmentation is required of IPv4 packets then the packet is punted to the LC CPU so no hardware assistance on fragemented packets (continuous fragmented flows will never reach wirerate). IPv6 fragmentation is not supported, the packet is punted and ICMP messages are generated back to the sender (by the LC CPU).
Previous page: ASR9000v / IOS-XRv
Next page: ASR9901 Specific Hardware Overview