ASR9001 Specific Hardware Overview

References:
http://www.cisco.com/c/en/us/products/collateral/routers/asr-9001-router/data_sheet_c78-685687.html
http://www.cisco.com/c/en/us/support/docs/routers/asr-9000-series-aggregation-services-routers/200135-Aggregation-Services-Router-9001-ASR-90.html#anc13
BRKARC-2003 - Cisco ASR9000 Architecture and Network Virtualisation
BRKSPG-2904 - ASR-9000 IOS-XR Hardware Architecture, QOS, EVC, IOS-XR Configuration and Troubleshooting
http://www.cisco.com/c/en/us/support/docs/routers/asr-9000-series-aggregation-services-routers/116999-problem-line-card-00.html

The ASR9001 "Iron Man" has 1 built in RSP with quad-core PPC CPU and 8GBs of RAM. The chassis supports 120Gbps of non-blocking full-duplex traffic (dual Typhoon chips). It has 4x 10Gbps SFP+s ports built in and 2x Modular Port Adapters (MPAs) bays that will support the 20x 1Gbps / 2x 10Gbps / 4x 10Gbps / 1x 40Gbps line cards. The internals of the ASR9001 (the NPUs and the built in 4x10G SFP+ ports) are built using the 2nd generation Typhoon line card hardware (which support four million IPv4 or two million IPv6 prefixes in shared TCAM by default, without changing the scaling profile, and one million MPLS labels).

Even though the ASR9001 supports OIR, all ports on the NP attached to the bay the new MPA is being inserted into will experience a brief outage (should be sub-second) as all ports go through a fast reset.

The ASR9001-S is the exact same unit as the ASR9001 but two of the built in 10G SFP+ ports and the 2nd MPA slot are disabled, but can be enabled with a license upgrade and reboot.

The 9001 chassis has two Network Processors Units (NPUs/NPs), so for the ASR9001-S the 2nd NP is disabled. The first MPA bay and 2 of the built in SFP+ ports are on one NP and the 2nd bay and remaining two built in SFP+ ports are on the other NP. These are Typhoon NPs that have 6x 10G lanes, so 60Gbps of non-blocking throughput per NPU, hense 120Gbps non-blocking per chassis.

The example ASR9001 chassis below has a 20x1G MPA in bay 0 (Gi0/0/0/0 to Gi0/0/0/19) and a 4x 10G MPA in bay 1 (Te0/0/1/0 to Te0/0/1/3). The built in SFP+ ports (Te0/0/2/0 to Te0/0/2/4) sit in "virtual" bay 2. Altogether, bay 0 (0/0/0/x) and bay 1 (0/0/1/x) sit on an internal Modular Line Card which shows as 0/0/CPU (like a MOD80-SE MPA). Bay 2 sits (0/0/2/x) sits on a Virtual Module and they all share the MPA same CPU, 0/0/CPU0. So the port numbers are Chassis/MPA/Bay/Port.

For comparison, in an ASR9006 for example with two A9K-MOD80-SE "80G Modular Linecard, Service Edge Optimized" MPAs, and a single 4x10G line card in one bay of each of the MOD80-SE MPAs, the first card shows as Te0/0/0/0 to Te0/0/0/3 and the second as Te0/1/0/0 to Te0/1/0/3:

RP/0/RSP0/CPU0:abr1#show inventory
Tue Dec 15 15:02:27.735 UTC
NAME: "module 0/RSP0/CPU0", DESCR: "ASR 9001, Route Switch Processor with 8GB memory"
PID: ASR9001-RP, VID: V01 , SN: xxxxx

NAME: "module 0/0/CPU0", DESCR: "ASR 9001, Modular Line Card"
PID: ASR9001-LC, VID: V01 , SN: xxxxxx

NAME: "module 0/0/0", DESCR: "ASR 9000 20-port 1GE Modular Port Adapter"
PID: A9K-MPA-20X1GE, VID: V02, SN: xxxxxx

NAME: "module 0/0/1", DESCR: "ASR 9000 4-port 10GE Modular Port Adapter"
PID: A9K-MPA-4X10GE, VID: V05, SN: xxxxx

NAME: "module mau 0/0/1/0", DESCR: "Multirate 10GBASE-LR and OC-192/STM-64 SR-1 XFP, SMF"
PID: XFP-10GLR-OC192SR   , VID: V04 , SN: xxxxx

NAME: "module 0/0/2", DESCR: "ASR 9000 Virtual Module"
PID: A9K-MODULEv, VID: N/A, SN: N/A

NAME: "module mau 0/0/2/0", DESCR: "10GBASE-LR SFP+ Module for SMF"
PID: SFP-10G-LR          , VID: V01 , SN: xxxx

NAME: "module mau 0/0/2/1", DESCR: "10GBASE-LR SFP+ Module for SMF"
PID: SFP-10G-LR          , VID: V01 , SN: xxxxx

NAME: "module mau 0/0/2/2", DESCR: "10GBASE-LR SFP+ Module for SMF"
PID: SFP-10G-LR          , VID: V02 , SN: xxxxx

NAME: "module mau 0/0/2/3", DESCR: "10GBASE-LR SFP+ Module for SMF"
PID: SFP-10G-LR          , VID: V01 , SN: xxxxxx


RP/0/RSP0/CPU0:abr1#show hw-module fpd location all
Tue Dec 15 14:03:27.705 UTC

===================================== ==========================================
                                      Existing Field Programmable Devices
                                      ==========================================
                                        HW                       Current SW Upg/
Location     Card Type                Version Type Subtype Inst   Version   Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/RSP0/CPU0  ASR9001-RP                 1.0   lc   fpga2   0       1.14     No
                                              lc   cbc     0      22.114    No
                                              lc   rommon  0       2.01     No
--------------------------------------------------------------------------------
0/FT0/SP     ASR-9001-FAN               1.0   ft   cbc     7      24.114    No
--------------------------------------------------------------------------------
0/0/CPU0     ASR9001-LC                 1.0   lc   cbc     0      23.114    No
                                              lc   fpga4   0       2.07     No
                                              lc   fpga2   0       1.18     No
                                              lc   rommon  0       2.01     No
--------------------------------------------------------------------------------
0/0/0        A9K-MPA-20X1GE             1.0   spa  fpga3   0       0.08     No
--------------------------------------------------------------------------------
0/0/1        A9K-MPA-4X10GE             1.0   spa  fpga6   1       1.06     No
--------------------------------------------------------------------------------


RP/0/RSP0/CPU0:abr1#show platform summary location all
Tue Dec 15 14:06:13.257 UTC
-------------------------------------------------------------------------------
     Platform Node : 0/RSP0/CPU0 (slot 0)
               PID : ASR9001-RP
         Card Type : ASR 9001, Route Switch Processor with 8GB memory
            VID/SN : V01  / xxxxxx
        Oper State : IOS XR RUN
        Last Reset : User Initiated reload                                                  Process: reload
                   : Tue Jan 27 00:27:55 2015
     Configuration : Power is enabled
                     Bootup enabled.
                     Monitoring enabled
        Rommon Ver : Version 2.01(20130627:172232)
        IOS SW Ver : 4.3.4
        Main Power : Power state Enabled. Estimate power 50 Watts of power required.
            Faults : N/A
-------------------------------------------------------------------------------
     Platform Node : 0/0/CPU0 (slot 2)
               PID : ASR9001-LC
         Card Type : ASR 9001, Modular Line Card
            VID/SN : V01  / xxxxx
        Oper State : IOS XR RUN
        Last Reset : MBI-HELLO powering down node on receiving powerdown notification       Process: mbi-hello
                   : Tue Mar  4 12:31:14 2014
     Configuration : Power is enabled
                     Bootup enabled.
                     Monitoring enabled
        Rommon Ver : Version 2.01(20130628:060031)
        IOS SW Ver : 4.3.4
        Main Power : Power state Enabled. Estimate power 330 Watts of power required.
            Faults : N/A
-------------------------------------------------------------------------------
     Platform Node : 0/0/0 (slot 2, bay 0)
               PID : A9K-MPA-20X1GE
         Card Type : ASR 9000 20-port 1GE Modular Port Adapter
            VID/SN : V02 / xxxxxx
        Oper State : OK
        Last Reset : Tue Jan 27 00:35:07 2015
           Up Time : 46w0d
     Configuration : Power is enabled
                     Bootup enabled
                     Monitoring enabled
     FPD SW Rev(s) :
-------------------------------------------------------------------------------
     Platform Node : 0/0/1 (slot 2, bay 1)
               PID : A9K-MPA-4X10GE
         Card Type : ASR 9000 4-port 10GE Modular Port Adapter
            VID/SN : V05 / xxxxxx
        Oper State : OK
        Last Reset : Tue Jan 27 00:34:49 2015
           Up Time : 46w0d
     Configuration : Power is enabled
                     Bootup enabled
                     Monitoring enabled
     FPD SW Rev(s) :
-------------------------------------------------------------------------------

The ASR9001 has two NPs. The output below shows which ports are mapped to which NP and which Fabric Interconnect ASIC:

RP/0/RSP0/CPU0:abr1#show controllers np summary all
Tue Dec 15 14:20:53.225 UTC

                Node: 0/0/CPU0:
----------------------------------------------------------------
 [total 2 NP]  Driver Version 12.46a Build 3
 NP 0 :  Hardware Revision: v2 A0
      :  Ucode - Version: 1.0 Build Date: ( Nov 9 2015, 20:50:00 )
 NP 1 :  Hardware Revision: v2 A0
      :  Ucode - Version: 1.0 Build Date: ( Nov 9 2015, 20:50:00 )


RP/0/RSP0/CPU0:abr1#show controllers np ports all
Tue Dec 15 14:21:52.537 UTC

                Node: 0/0/CPU0:
----------------------------------------------------------------

NP Bridge Fia                       Ports
-- ------ --- ---------------------------------------------------
0  0      0   GigabitEthernet0/0/0/0 - GigabitEthernet0/0/0/19
0  0      0   TenGigE0/0/2/0, TenGigE0/0/2/1
1  1      1   TenGigE0/0/1/0, TenGigE0/0/1/1, TenGigE0/0/1/2, TenGigE0/0/1/3
1  1      1   TenGigE0/0/2/2, TenGigE0/0/2/3



RP/0/RSP0/CPU0:abr1#show controllers np portMap all
Tue Dec 15 14:31:51.914 UTC

                Node: 0/0/CPU0:
----------------------------------------------------------------

Show Port Map for NP: 0, and RX Unicast Ports
phy port num      interface desc                  uiMappedSourcePort
0                                                 UNDEFINED
1                                                 UNDEFINED
2                                                 UNDEFINED
3                                                 UNDEFINED
4                                                 UNDEFINED
5                                                 UNDEFINED
6                                                 UNDEFINED
7                                                 UNDEFINED
8                 TenGigE0/0/2/0                  63 (unmapped)
9                 TenGigE0/0/2/1                  63 (unmapped)
10                                                UNDEFINED
11                Egress Fabric1                  63 (unmapped)
12                Egress Fabric2                  63 (unmapped)
13                                                UNDEFINED
14                                                UNDEFINED
15                                                UNDEFINED
16                GigE0/0/0/0                     63 (unmapped)
17                GigE0/0/0/1                     63 (unmapped)
18                GigE0/0/0/2                     63 (unmapped)
19                GigE0/0/0/3                     63 (unmapped)
20                GigE0/0/0/8                     63 (unmapped)
21                GigE0/0/0/9                     63 (unmapped)
22                Unknown                         63 (unmapped)
23                Unknown                         63 (unmapped)
24                GigE0/0/0/10                    63 (unmapped)
25                GigE0/0/0/11                    63 (unmapped)
26                GigE0/0/0/12                    63 (unmapped)
27                GigE0/0/0/13                    63 (unmapped)
28                GigE0/0/0/18                    63 (unmapped)
29                GigE0/0/0/19                    63 (unmapped)
30                Unknown                         63 (unmapped)
31                Unknown                         63 (unmapped)
32                                                UNDEFINED
33                                                UNDEFINED
34                                                UNDEFINED
35                                                UNDEFINED
36                                                UNDEFINED
37                                                UNDEFINED
38                                                UNDEFINED
39                                                UNDEFINED
40                GigE0/0/0/4                     63 (unmapped)
41                GigE0/0/0/5                     63 (unmapped)
42                GigE0/0/0/6                     0  (bundle)
43                GigE0/0/0/7                     0  (bundle)
44                                                UNDEFINED
45                                                UNDEFINED
46                                                UNDEFINED
47                                                UNDEFINED
48                GigE0/0/0/14                    63 (unmapped)
49                GigE0/0/0/15                    63 (unmapped)
50                GigE0/0/0/16                    63 (unmapped)
51                GigE0/0/0/17                    63 (unmapped)
52                                                UNDEFINED
53                                                UNDEFINED
54                                                UNDEFINED
55                                                UNDEFINED
56                                                UNDEFINED
57                                                UNDEFINED
58                                                UNDEFINED
59                                                UNDEFINED
60                                                UNDEFINED
61                                                UNDEFINED
62                                                UNDEFINED
63                                                UNDEFINED

Show Port Map for NP: 1, and RX Unicast Ports
phy port num      interface desc                  uiMappedSourcePort
0                                                 UNDEFINED
1                                                 UNDEFINED
2                                                 UNDEFINED
3                                                 UNDEFINED
4                 TenGigE0/0/1/0                  63 (unmapped)
5                 TenGigE0/0/1/1                  63 (unmapped)
6                 TenGigE0/0/1/2                  63 (unmapped)
7                 TenGigE0/0/1/3                  63 (unmapped)
8                 TenGigE0/0/2/2                  63 (unmapped)
9                 TenGigE0/0/2/3                  63 (unmapped)
10                                                UNDEFINED
11                Egress Fabric1                  63 (unmapped)
12                Egress Fabric2                  63 (unmapped)
13                                                UNDEFINED
14                                                UNDEFINED
15                                                UNDEFINED
16                                                UNDEFINED
17                                                UNDEFINED
18                                                UNDEFINED
19                                                UNDEFINED
20                                                UNDEFINED
21                                                UNDEFINED
22                                                UNDEFINED
23                                                UNDEFINED
24                                                UNDEFINED
25                                                UNDEFINED
26                                                UNDEFINED
27                                                UNDEFINED
28                                                UNDEFINED
29                                                UNDEFINED
30                                                UNDEFINED
31                                                UNDEFINED
32                                                UNDEFINED
33                                                UNDEFINED
34                                                UNDEFINED
35                                                UNDEFINED
36                                                UNDEFINED
37                                                UNDEFINED
38                                                UNDEFINED
39                                                UNDEFINED
40                                                UNDEFINED
41                                                UNDEFINED
42                                                UNDEFINED
43                                                UNDEFINED
44                                                UNDEFINED
45                                                UNDEFINED
46                                                UNDEFINED
47                                                UNDEFINED
48                                                UNDEFINED
49                                                UNDEFINED
50                                                UNDEFINED
51                                                UNDEFINED
52                                                UNDEFINED
53                                                UNDEFINED
54                                                UNDEFINED
55                                                UNDEFINED
56                                                UNDEFINED
57                                                UNDEFINED
58                                                UNDEFINED
59                                                UNDEFINED
60                                                UNDEFINED
61                                                UNDEFINED
62                                                UNDEFINED
63                                                UNDEFINED

The ASR-9001 has a single Fabric Switching ASIC (Sacramento) at location 0/0/CPU0 and two Fabric Interface ASIC (FIAs). The Fabric Switching ASIC (Skytrain) is the same ASIC that is used on the RSP440 and 2nd generation based linecards on other ASR9000 platforms. The Fabric Switching ASIC has 4 ports with 2 connected to each FIA, each FIA connects to one of the two Trident NPUs and like the NPUs has 60Gbps of non-blocking throughput capacity. The following command displays the counters related to the Fabric Switching ASIC:

RP/0/RSP0/CPU0:abr1#show controllers fabric crossbar statistics instance 0 location 0/0/CPU0
Tue Dec 15 15:13:54.152 UTC
Port statistics for xbar:0 port:1
==============================
Hi priority stats (unicast)
===========================
    Ingress Packet Count Since Last Read       : 1673157979
    Egress Packet Count Since Last Read        : 1026151308

Low priority stats (multicast)
===========================
    Ingress Packet Count Since Last Read       : 11895

Port statistics for xbar:0 port:2
==============================
Hi priority stats (unicast)
===========================
    Ingress Packet Count Since Last Read       : 1677497411
    Egress Packet Count Since Last Read        : 1026151307

Low priority stats (multicast)
===========================
    Ingress Packet Count Since Last Read       : 16856
    Egress Packet Count Since Last Read        : 2595832

Port statistics for xbar:0 port:3
==============================
Hi priority stats (unicast)
===========================
    Ingress Packet Count Since Last Read       : 6568112590
    Ingress Channel Utilization Count          : 2
    Egress Packet Count Since Last Read        : 7209854261
    Egress Channel Utilization Count           : 2

Low priority stats (multicast)
===========================
    Ingress Packet Count Since Last Read       : 2714700
    Egress Packet Count Since Last Read        : 2743723

Port statistics for xbar:0 port:4
==============================
Hi priority stats (unicast)
===========================
    Ingress Packet Count Since Last Read       : 6553243164
    Ingress Channel Utilization Count          : 2
    Egress Packet Count Since Last Read        : 7209854283
    Egress Channel Utilization Count           : 2

Low priority stats (multicast)
===========================
    Ingress Packet Count Since Last Read       : 148481
    Egress Packet Count Since Last Read        : 148208


Total Unicast In:    16472011144
Total Unicast Out:   16472011159
Total Multicast In:  2891932
Total Multicast Out: 5487763



RP/0/RSP0/CPU0:abr1#show controllers fabric crossbar link-status instance 0 location 0/0/CPU0
Tue Dec 15 15:20:21.033 UTC
PORT    Remote Slot  Remote Inst    Logical ID  Status
======================================================
01        02               01             1        Up
02        02               01             0        Up
03        02               00             0        Up
04        02               00             1        Up

The following commands shows the counters for the FIAs

RP/0/RSP0/CPU0:abr1#show controllers fabric fia link-status location 0/0/CPU0
Tue Dec 15 15:19:35.577 UTC

 ********** FIA-0 **********
Category: link-0
                                 spaui link-0                       Up
                                 spaui link-1                       Up
                                  arb  link-0                       Up
                                  xbar link-0                       Up
                                  xbar link-1                       Up

 ********** FIA-1 **********
Category: link-1
                                 spaui link-0                       Up
                                 spaui link-1                       Up
                                  arb  link-0                       Up
                                  xbar link-0                       Up
                                  xbar link-1                       Up



RP/0/RSP0/CPU0:abr1#show controllers fabric fia stats location 0/0/CPU0
Tue Dec 15 15:15:50.506 UTC

 ********** FIA-0 **********
Category: count-0
                         From Unicast Xbar[0]               7228789172
                         From Unicast Xbar[1]               7228789179
                         From Unicast Xbar[2]                        0
                         From Unicast Xbar[3]                        0
                       From MultiCast Xbar[0]                  2754474
                       From MultiCast Xbar[1]                   148488
                       From MultiCast Xbar[2]                        0
                       From MultiCast Xbar[3]                        0
                           To Unicast Xbar[0]               6585755334
                           To Unicast Xbar[1]               6570846462
                           To Unicast Xbar[2]                        0
                           To Unicast Xbar[3]                        0
                         To MultiCast Xbar[0]                  2725357
                         To MultiCast Xbar[1]                   148812
                         To MultiCast Xbar[2]                        0
                         To MultiCast Xbar[3]                        0
                         To Line Interface[0]               3607130880
                         To Line Interface[1]              10945907385
                       From Line Interface[0]               3693656345
                       From Line Interface[1]               9530309407
                                Ingress drop:                        6
                                 Egress drop:                        0
                                 Total drop:                         6

 ********** FIA-1 **********
Category: count-1
                         From Unicast Xbar[0]               1029084883
                         From Unicast Xbar[1]               1029084885
                         From Unicast Xbar[2]                        0
                         From Unicast Xbar[3]                        0
                       From MultiCast Xbar[0]                  2606325
                       From MultiCast Xbar[1]                        0
                       From MultiCast Xbar[2]                        0
                       From MultiCast Xbar[3]                        0
                           To Unicast Xbar[0]               1681749349
                           To Unicast Xbar[1]               1677400227
                           To Unicast Xbar[2]                        0
                           To Unicast Xbar[3]                        0
                         To MultiCast Xbar[0]                    16882
                         To MultiCast Xbar[1]                    11912
                         To MultiCast Xbar[2]                        0
                         To MultiCast Xbar[3]                        0
                         To Line Interface[0]               1383470551
                         To Line Interface[1]                680999648
                       From Line Interface[0]               2792166675
                       From Line Interface[1]                601244228
                                Ingress drop:                        7
                                 Egress drop:                        0
                                 Total drop:                         7

One can see errors for all ASICs with the following:

RP/0/RSP0/CPU0:abr1-lhclon.core#show asic-errors all location 0/0/CPU0
Tue Dec 15 16:08:31.297 UTC
************************************************************
*                  Fia ASIC Error Summary                  *
************************************************************
Instance            : 0
Number of nodes     : 0
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
Instance            : 1
Number of nodes     : 0
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------

************************************************************
*                Prm_np ASIC Error Summary                 *
************************************************************
Instance            : 0
Number of nodes     : 0
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
Instance            : 1
Number of nodes     : 0
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------

The hardware packet flow is as follows:

- For MPA cards: MPA Port PHYs > NP > FIA > FSA > to FIA on another MPA > NP > PHY
- For the built int SFP+ ports: SFP+ PHY > direct to NP > FIA > FSA > FIA > NP > PHY
- There is 40Gbps of bandwidth available to each bay from each NPU (20Gbps is used by the 2 built in SFP+ ports assigned to each NPU) so the chassis cannot be oversubscribed
- Each Line Card has a LC CPU accessed via the NP (PHY > NP > LC CPU) for features like programming hardware, inline NetFlow, SW switched packets

The NP contains the lookup table (FIB, MAC), TCAM, Stats memory, Frame memory and performs LPTS (CoPP). Each line card has a CPU that provides local logic to run control plane protocols like BFD, ARP, ICMP, OAM NetFlow etc all distributed at the line-card level.

 

 

Larger routers and line cards have a bridges that sits between the NPs and FIAs, it sits as a non-blocking memory converter between the two and can be checked using "show controllers fabric fia bridge stats location 0/RSP0/CPU0". On the 9001 there is one bridge per NP and they cannot be interacted with.

 

Down at the interface level, the four built-in SFP+ interface are 10G only, not mixed 1/10G. If fragmentation is required of IPv4 packets then the packet is punted to the LC CPU so no hardware assistance on fragemented packets (continuous fragmented flows will never reach wirerate). IPv6 fragmentation is not supported, the packet is punted and ICMP messages are generated back to the sender (by the LC CPU).