Date created: 11/02/21 10:44:34. Last modified: 01/14/22 14:52:43

NCS540 Troubleshooting

 

NPU Debugging

RP/0/RP0/CPU0:ma0#show controllers npu diag counters graphical instance 0 location 0/0/CPU0
Tue Nov  2 10:08:00.621 GMT

                                            |                                                                      /|\
                                            |                   Q A X   N E T W O R K   I N T E R F A C E           |
                                           \|/                                                                      |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|                                                                                      NBI                                                                                      |
|  RX_TOTAL_BYTE_COUNTER                         = 2,490,699                            |  TX_TOTAL_BYTE_COUNTER                         = 3,929,616                            |
|  RX_TOTAL_PKT_COUNTER                          = 9,549                                |  TX_TOTAL_PKT_COUNTER                          = 13,160                               |
|  RX_TOTAL_DROPPED_EOPS                         = 0                                    |                                                                                       |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|                                          IRE                                          |                                         EPNI                                          |
|  CPU_PACKET_COUNTER                            = 0                                    |                                                                                       |
|  NIF_PACKET_COUNTER                            = 9,549                                |  EPE_BYTES_COUNTER                             = 4,271,899                            |
|  OAMP_PACKET_COUNTER                           = 1,036                                |  EPE_PKT_COUNTER                               = 16,363                               |
|  OLP_PACKET_COUNTER                            = 3                                    |  EPE_DSCRD_PKT_CNT                             = 52                                   |
|  IRE_FDT_INTRFACE_CNT                          = 0                                    |                                                                                       |
|  RCY_PACKET_COUNTER                            = 4,896                                |                                                                                       |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|                                          DDP                                          |                                          EGQ                                          |
|                                                                                       |                                                                                       |
|  MMU_RD_REQ_CNT                                = 0                                    |                                                                                       |
+-------------------------------------------+-------------------------------------------|                                                                                       |
|                                          SPB                                          |
|  SPB_AF_SRAM_RJCT_PKT_CNT                      = 0                                    |  PQP_DSCRD_UC_PKT_CNT                          = 0                                    |
|  SPB_AF_N_SRAM_RJCT_PKT_CNT                    = 0                                    |  PQP_UC_BYTES_CNT                              = 4,686,689                            |
+-------------------------------------------+-------------------------------------------|  PQP_MC_PKT_CNT                                = 3                                    |
|                                          CGM                                          |  PQP_DSCRD_MC_PKT_CNT                          = 81                                   |
|                                                                                       |  PQP_MC_BYTES_CNT                              = 228                                  |
|  CGM_VOQ_SRAM_ENQ_PKT_CTR                      = 16,498                               |  EHP_UNICAST_PKT_CNT                           = 16,412                               |
|  CGM_VOQ_SRAM_DEQ_PKT_CTR                      = 16,498                               |  EHP_MC_HIGH_PKT_CNT                           = 0                                    |
|  CGM_VOQ_SRAM_DEL_PKT_CTR                      = 0                                    |  EHP_MC_LOW_PKT_CNT                            = 3                                    |
|  CGM_VOQ_SRAM_ENQ_RJCT_PKT_CTR                 = 3,057                                |  PRP_PKT_DSCRD_TDM_CNT                         = 0                                    |
|  CGM_VOQ_DRAM_ENQ_BUNDLE_CTR                   = 0                                    |  PRP_SOP_DSCRD_UC_CNT                          = 0                                    |
|  CGM_VOQ_DRAM_DEQ_PKT_CTR                      = 0                                    |  PRP_SOP_DSCRD_MC_CNT                          = 0                                    |
|  CGM_VOQ_DRAM_DEL_PKT_CTR                      = 0                                    |  PRP_SOP_DSCRD_TDM_CNT                         = 0                                    |
|                                                                                       |  DELETED_PKT_CNT                               = 0                                    |
|   Rejects: DP_LEVEL EXTERNAL_ERR QNUM_NOT_VALID                                       |   Discards: IPV4_TTL_EQUALS_ONE SRC_EQUAL_DEST                                        |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|                                                                                       |                                          FDA                                          |
|                                          TXQ                                          |  CELLS_IN_TDM_CNT        = 0              |  CELLS_OUT_TDM_CNT       = 0              |
|                                                                                       |  CELLS_IN_MESHMC_CNT     = 0              |  CELLS_OUT_MESHMC_CNT    = 0              |
|  EGQ_SRAM_PKT_CNT        = 50,888        -|  EGQ_DRAM_PKT_CNT        = 0             --> CELLS_IN_IPT_CNT        = 28,110         |  CELLS_OUT_IPT_CNT       = 28,110         |
|  FDT_SRAM_PKT_CNT        = 0              |  FDT_DRAM_PKT_CNT        = 0              |                                                                                       |
|                                                                                       |  EGQ_TDM_OVF_DROP_CNT                          = 0                                    |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|                                          FDT                                          |                                          FDR                                          |
|  IPT_DESC_CELL_COUNTER                         = 0                                    |  P1_CELL_IN_CNT                                = 0                                    |
|  IRE_DESC_CELL_COUNTER                         = 0                                    |  P2_CELL_IN_CNT                                = 0                                    |
|                                                                                       |  P3_CELL_IN_CNT                                = 0                                    |
|  TRANSMITTED_DATA_CELLS_COUNTER                = 0                                    |  CELL_IN_CNT_TOTAL                             = 0                                    |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
                                            |                                                                      /|\
                                            |                    Q A X   F A B R I C   I N T E R F A C E            |
                                           \|/                                                                      |

RP/0/RP0/CPU0:ma0#show controllers npu diag pp ?
  EncapsulationInfo     Dump last packet encapsulation detail(cisco-support)
  ForwardDecisionTrace  Basic forwarding information for last ingress frame(cisco-support)
  ParsingInfo           Dump last packet detail from parser(cisco-support)
  PktAssociatedTmInfo   Dump last packet Forwarding id detail(cisco-support)
  ReceivedPacketInfo    Dump last recieved packet detail(cisco-support)
  TerminationInfo       List of programs used for termination(cisco-support)
  TrapsInfo             Last Trap information(cisco-support)


#show controllers fia statistics detail instance all location all


RP/0/RP0/CPU0:ma0#show controllers npu resources ?
  all               all hardware resources
  ecmpfec           ecmp_fec
  encap             encap
  exttcamipv4       ext_tcam_ipv4
  exttcamipv6       ext_tcam_ipv6
  exttcamipv6long   ext_tcam_ipv6_long
  exttcamipv6short  ext_tcam_ipv6_short
  fec               fec
  lem               lem
  lpm               lpm
  stats             Show stats hardware resources(cisco-support)



RP/0/RP0/CPU0:ma0#show controllers npu stats traps-all instance 0 location 0/0/CPU0
Tue Nov  2 10:16:28.203 GMT


Trap Type                                     NPU  Trap TrapStats   Policer Packet               Packet

                                              ID    ID      ID              Accepted             Dropped
============================================================================================================================
RxTrapMimDiscardMacsaDrop (IRB)               0    1    0x1         32045   0                    0
RxTrapMimDiscardMacsaTrap (ERP_BDL)           0    2    0x2         32041   0                    0
RxTrapMimDiscardMacsaSnoop(dot1x)             0    3    0x3         32020   0                    0
RxTrapMimTeSaMove(SPIO)                       0    4    0x4         32041   6434471              0
RxTrapMimSaMove(CFM_DOWM_MEP_DMM)             0    6    0x6         32039   0                    0
RxTrapMimSaUnknown(RCY_CFM_DOWN_MEP_DMM)      0    7    0x7         32039   0                    0
RxTrapAuthSaLookupFail (IPMC default)         0    8    0x8         32035   0                    0
RxTrapAuthSaPortFail (L3 wrong MAC)           0    9    0x9         32020   17                   0
RxTrapAuthSaVlanFail (L3 unknown-MC/BC)       0    10   0xa         32020   95                   5
RxTrapSaMulticast                             0    11   0xb         32020   1                    0
bcmRxTrapTerminatedSaMulticast                0    12   0xc         32020   0                    0
RxTrapArpMyIp                                 0    14   0xe         32001   0                    0
RxTrapArp                                     0    15   0xf         32001   51                   0
RxTrapDhcpv4Server                            0    19   0x13        32019   0                    0
RxTrapDhcpv4Client                            0    20   0x14        32024   0                    0
RxTrapDhcpv6Server                            0    21   0x15        32019   0                    0
RxTrapDhcpv6Client                            0    22   0x16        32024   0                    0
RxTrapL2Cache_LACP                            0    24   0x18        32003   0                    0
RxTrapL2Cache_LLDP1                           0    25   0x19        32004   0                    0
RxTrapL2Cache_LLDP2                           0    26   0x1a        32004   982956               0
RxTrapL2Cache_ELMI                            0    28   0x1c        32005   0                    0
RxTrapL2Cache_BPDU                            0    29   0x1d        32029   0                    0
RxTrapL2Cache_BUNDLE_BPDU                     0    30   0x1e        32029   0                    0
RxTrapL2Cache_CDP                             0    31   0x1f        32002   12                   0
RxTrapHeaderSizeErr                           0    33   0x21        32020   0                    0
bcmRxTrapTerminatedHeaderSizeErr              0    34   0x22        32020   0                    0
RxTrapIpCompMcInvalidIp                       0    37   0x25        32020   0                    0
RxTrapMyMacAndIpDisabled                      0    38   0x26        32020   7                    1
RxTrapMyMacAndMplsDisable                     0    39   0x27        32020   0                    0
RxTrapArpReply                                0    40   0x28        32001   1364                 0
RxTrapFibDrop                                 0    43   0x2b        32020   0                    0
RxTrapMTU                                     0    44   0x2c        32022   0                    0
RxTrapMiscDrop                                0    45   0x2d        32020   0                    0
RxTrapL2AclDeny                               0    46   0x2e        32036   0                    0
Rx_UNKNOWN_PACKET                             0    48   0x30        32020   80882737             37765167
RxTrapL3AclDeny                               0    49   0x31        32036   0                    0
RxTrapStpStateBlock_EXT_HDR                   0    51   0x33        32048   0                    0
RxTrapFailover1Plus1Fail                      0    55   0x37        32043   0                    0
RxTrapOamY1731MplsTp(OAM_SWOFF_DN_CCM)        0    59   0x3b        32031   0                    0
RxTrapOamY1731Pwe(OAM_SWOFF_UP_CCM)           0    60   0x3c        32032   0                    0
RxTrapOamLevel                                0    67   0x43        32025   0                    0
RxTrapRedirectToCpuOamPacket                  0    68   0x44        32027   0                    0
RxTrapOamPassive                              0    69   0x45        32026   0                    0
RxTrap1588                                    0    70   0x46        32040   0                    0
RxTrapExternalLookupError                     0    71   0x47        32020   0                    0
RxTrapArplookupFail                           0    72   0x48        32001   0                    0
RxTrapL2cpDrop (INVALID L2CP)                 0    74   0x4a        32046   0                    0
RxTrapTrillUnknownUc (flooding UC disable)    0    81   0x51        32044   0                    0
RxTrapTrillUnknownMc (flooding MC-BC disable) 0    82   0x52        32044   0                    0
RxTrapMplsControlWordTrap                     0    86   0x56        32017   0                    0
bcmRxTrapTerminatedMplsControlWordTrap        0    87   0x57        32017   0                    0
RxTrapMplsControlWordDrop                     0    88   0x58        32017   0                    0
RxTrapMplsUnknownLabel                        0    90   0x5a        32020   57293188             6651677
RxTrapIpv4VersionError                        0    98   0x62        32020   8                    0
bcmRxTrapTerminatedIpv4VersionError           0    99   0x63        32020   0                    0
RxTrapIpv4ChecksumError                       0    100  0x64        32020   0                    0
bcmRxTrapTerminatedIpv4ChecksumError          0    101  0x65        32020   0                    0
RxTrapIpv4HeaderLengthError                   0    102  0x66        32020   0                    0
bcmRxTrapTerminatedIpv4HeaderLengthError      0    103  0x67        32020   0                    0
RxTrapIpv4TotalLengthError                    0    104  0x68        32020   0                    0
bcmRxTrapTerminatedIpv4TotalLengthError       0    105  0x69        32020   0                    0
RxTrapIpv4Ttl0                                0    106  0x6a        32010   0                    0
bcmRxTrapTerminatedIpv4Ttl0                   0    107  0x6b        32010   0                    0
RxTrapIpv4Ttl1                                0    110  0x6e        32010   317670               0
bcmRxTrapTerminatedIpv4Ttl1                   0    111  0x6f        32010   0                    0
RxTrapIpv4DipZero                             0    113  0x71        32020   0                    0
bcmRxTrapTerminatedIpv4DipZero                0    114  0x72        32020   0                    0
RxTrapIpv4SipIsMc                             0    115  0x73        32020   0                    0
bcmRxTrapTerminatedIpv4SipIsMc                0    116  0x74        32020   0                    0
RxTrapIpv6VersionError                        0    118  0x76        32020   0                    0
bcmRxTrapTerminatedIpv6VersionError           0    119  0x77        32020   0                    0
RxTrapIpv6HopCount0                           0    120  0x78        32013   0                    0
RxTrapIpv6LoopbackAddress                     0    123  0x7b        32020   0                    0
bcmRxTrapTerminatedIpv6LoopbackAddress        0    124  0x7c        32020   0                    0
RxTrapIpv6MulticastSource                     0    125  0x7d        32020   0                    0
bcmRxTrapTerminatedIpv6MulticastSource        0    126  0x7e        32020   0                    0
RxTrapIpv6NextHeaderNull                      0    127  0x7f        32012   0                    0
bcmRxTrapTerminatedIpv6NextHeaderNull         0    128  0x80        32012   0                    0
RxTrapSRv6Cnt3                                0    131  0x83        32020   0                    0
bcmRxTrapTerminatedIpv6LocalSiteDestination   0    132  0x84        32020   0                    0
RxTrapIpv6Ipv4CompatibleDestination           0    135  0x87        32020   0                    0
bcmRxTrapTerminatedIpv6Ipv4CompatibleDestination 0    136  0x88        32020   0                    0
RxTrapMplsTtl0                                0    139  0x8b        32014   0                    0
RxTrapMplsTtl1                                0    140  0x8c        32014   91                   0
RxTrapMplsTtl1Terminated                      0    141  0x8d        32014   0                    0
RxTrapUcStrictRpfFail                         0    153  0x99        32037   0                    0
bcmRxTrapUcLooseRpfFail                       0    154  0x9a        32037   0                    0
RxTrapMcExplicitRpfFail                       0    155  0x9b        32035   0                    0
RxTrapOamp(OAM_BDL_DN_NON_CCM)                0    158  0x9e        32033   0                    0
RxTrapOamEthUpAccelerated(OAM_BDL_UP_NON_CCM) 0    162  0xa2        32034   0                    0
RxTrapReceive                                 0    167  0xa7        32019   272344642            0
RxTrapUserDefine_FIB_IPV4_NULL0               0    168  0xa8        32020   0                    0
RxTrapUserDefine_SR_Unknown_Label             0    169  0xa9        32020   43                   315
RxTrapUserDefine_FIB_IPV6_NULL0               0    170  0xaa        32020   0                    0
RxTrapUserDefine_FIB_IPV4_GLEAN               0    171  0xab        32018   83                   7791
RxTrapUserDefine_FIB_IPV6_GLEAN               0    172  0xac        32018   0                    0
RxTrapUserDefine_IPV4_OPTIONS                 0    173  0xad        32006   0                    0
RxTrapUserDefine_IPV4_RSVP_OPTIONS            0    174  0xae        32007   0                    0
RxTrapUserDefine                              0    175  0xaf        32028   0                    0
RxTrapUserDefine_IPV4_PIM_OPTIONS             0    177  0xb1        32008   0                    0
RxTrapUserDefine_RECEIVE_L2                   0    178  0xb2        32019   2665571              0
RxNetflowSnoopTrap0                           0    202  0xca        32020   0                    0
RxNetflowSnoopTrap1                           0    203  0xcb        32020   0                    0
bcmRxTrapUnknownDest                          0    207  0xcf        32020   0                    0
RxTrapSrv6EndOpPunt                           0    208  0xd0        32047   0                    0
bcmRxTrapMacSecurity                          0    209  0xd1        32049   0                    0
RxTrapUserDefine_OAM_SAT                      0    212  0xd4        32050   0                    0


# Some of the counters in "show controllers npu stats counters-all detail" have ambiguous names.
# Example below:

RP/0/RP0/CPU0:ma0# show controllers npu stats counters-all detail instance 3 location 0/3/CPU0 | inc "Drop|Rjct|Dscrd|Err|Del" | ex ": 0"
Fri Nov 12 11:22:53.514 GMT
IQM0 IqmCntCmdErrorsFilterA : 255
IQM0 IqmCntCmdErrorsFilterB : 255
IQM0 IrppCntCmdErrorsFilterA : 255
IQM0 IrppCntCmdErrorsFilterB : 255
IQM1 IqmCntCmdErrorsFilterA : 255
IQM1 IqmCntCmdErrorsFilterB : 255
IQM1 IrppCntCmdErrorsFilterA : 255
IQM1 IrppCntCmdErrorsFilterB : 255
NBIH BistRxErrLengthPktCnt : 15528
NBIH BistRxErrPktIndexCnt : 59950
NBIH BistRxErrDataCnt : 11002
NBIH BistRxErrInCrcErrCnt : 56425
NBIH BistRxErrSobCnt : 34699

Regarding BIST counters:

BistRxErrPktIndexCnt
BistRxErrDataCnt
BistRxErrInCrcErrCnt
BistRxErrSobCnt

"These error counters are harmless and can be ignored. These are caused by NBI BIST. Nbi Bist function is intended as a traffic generator & checker when the external port is looped back. But, unfortunately, the checker is running all the time, even when the traffic generator is not being used. The only time the counts are reliable is if the port is in external loopback, and traffic is coming from the NBI BIST traffic generator, rest of the time those numbers are invalid. These cannot be disabled too."


StEgrBillDeqRprtDropCtr StIngBillDeqRprtDropCtr StIngBillEnqRprtDropCtr "All these 3 drop counters above are harmless ..they can be ignored."
ErppLagPruningDscrdsCnt "This counter is expected in cases where lag has more than 1 members and multicast traffic needs replication on lag port. In that case, only one lag member will be chosen and rest of the members will drop the packet due to lag pruning. in such case, above counter will increment."
EhpDscrdPktCnt EpeDscrdPktsCnt "These are EGQ drops (inside the Egress Traffic Manager block). They may be related to egress congestion ..you should check “diag egr_congestion” output multiple times when these drops are incrementing."


# Remember that with the following command, the packet/byte counts are for traffic leaving the named interface (Te0/0/0/24).
# The command is showing traffic that ingressed an VoQ which will egress via the named interfaces. So despite saying "ingress"
# this command is showing egress traffic:
#
RP/0/RP0/CPU0:ma0#show controllers npu stats voq ingress interface Te0/0/0/24 instance 0 location 0/0/CPU0 Tue Nov 2 10:24:47.723 GMT Interface Name = Te0/0/0/24 Interface Handle = 48 Asic Instance = 0 VOQ Base = 1216 Port Speed(kbps) = 10000000 Local Port = local ReceivedPkts ReceivedBytes DroppedPkts DroppedBytes ------------------------------------------------------------------- TC_0 = 15 1500 0 0 TC_1 = 0 0 0 0 TC_2 = 0 0 0 0 TC_3 = 218765888 376013039231 0 0 TC_4 = 0 0 0 0 TC_5 = 0 0 0 0 TC_6 = 0 0 0 0 TC_7 = 0 0 0 0

 


Previous page: NCS540 Packet Capture
Next page: FIA Packet Diag