Date created: Tuesday, November 2, 2021 10:44:34 AM. Last modified: Wednesday, June 8, 2022 2:23:01 PM
NCS540 Troubleshooting
TCAM Stats
show controllers npu internaltcam location 0/0/CPU0
show controllers npu resources stats instance all location 0/0/CPU0
show controllers npu resources all location all
NPU Debugging
RP/0/RP0/CPU0:ma0#show controllers npu diag counters graphical instance 0 location 0/0/CPU0
Tue Nov 2 10:08:00.621 GMT
| /|\
| Q A X N E T W O R K I N T E R F A C E |
\|/ |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| NBI |
| RX_TOTAL_BYTE_COUNTER = 2,490,699 | TX_TOTAL_BYTE_COUNTER = 3,929,616 |
| RX_TOTAL_PKT_COUNTER = 9,549 | TX_TOTAL_PKT_COUNTER = 13,160 |
| RX_TOTAL_DROPPED_EOPS = 0 | |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| IRE | EPNI |
| CPU_PACKET_COUNTER = 0 | |
| NIF_PACKET_COUNTER = 9,549 | EPE_BYTES_COUNTER = 4,271,899 |
| OAMP_PACKET_COUNTER = 1,036 | EPE_PKT_COUNTER = 16,363 |
| OLP_PACKET_COUNTER = 3 | EPE_DSCRD_PKT_CNT = 52 |
| IRE_FDT_INTRFACE_CNT = 0 | |
| RCY_PACKET_COUNTER = 4,896 | |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| DDP | EGQ |
| | |
| MMU_RD_REQ_CNT = 0 | |
+-------------------------------------------+-------------------------------------------| |
| SPB |
| SPB_AF_SRAM_RJCT_PKT_CNT = 0 | PQP_DSCRD_UC_PKT_CNT = 0 |
| SPB_AF_N_SRAM_RJCT_PKT_CNT = 0 | PQP_UC_BYTES_CNT = 4,686,689 |
+-------------------------------------------+-------------------------------------------| PQP_MC_PKT_CNT = 3 |
| CGM | PQP_DSCRD_MC_PKT_CNT = 81 |
| | PQP_MC_BYTES_CNT = 228 |
| CGM_VOQ_SRAM_ENQ_PKT_CTR = 16,498 | EHP_UNICAST_PKT_CNT = 16,412 |
| CGM_VOQ_SRAM_DEQ_PKT_CTR = 16,498 | EHP_MC_HIGH_PKT_CNT = 0 |
| CGM_VOQ_SRAM_DEL_PKT_CTR = 0 | EHP_MC_LOW_PKT_CNT = 3 |
| CGM_VOQ_SRAM_ENQ_RJCT_PKT_CTR = 3,057 | PRP_PKT_DSCRD_TDM_CNT = 0 |
| CGM_VOQ_DRAM_ENQ_BUNDLE_CTR = 0 | PRP_SOP_DSCRD_UC_CNT = 0 |
| CGM_VOQ_DRAM_DEQ_PKT_CTR = 0 | PRP_SOP_DSCRD_MC_CNT = 0 |
| CGM_VOQ_DRAM_DEL_PKT_CTR = 0 | PRP_SOP_DSCRD_TDM_CNT = 0 |
| | DELETED_PKT_CNT = 0 |
| Rejects: DP_LEVEL EXTERNAL_ERR QNUM_NOT_VALID | Discards: IPV4_TTL_EQUALS_ONE SRC_EQUAL_DEST |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| | FDA |
| TXQ | CELLS_IN_TDM_CNT = 0 | CELLS_OUT_TDM_CNT = 0 |
| | CELLS_IN_MESHMC_CNT = 0 | CELLS_OUT_MESHMC_CNT = 0 |
| EGQ_SRAM_PKT_CNT = 50,888 -| EGQ_DRAM_PKT_CNT = 0 --> CELLS_IN_IPT_CNT = 28,110 | CELLS_OUT_IPT_CNT = 28,110 |
| FDT_SRAM_PKT_CNT = 0 | FDT_DRAM_PKT_CNT = 0 | |
| | EGQ_TDM_OVF_DROP_CNT = 0 |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| FDT | FDR |
| IPT_DESC_CELL_COUNTER = 0 | P1_CELL_IN_CNT = 0 |
| IRE_DESC_CELL_COUNTER = 0 | P2_CELL_IN_CNT = 0 |
| | P3_CELL_IN_CNT = 0 |
| TRANSMITTED_DATA_CELLS_COUNTER = 0 | CELL_IN_CNT_TOTAL = 0 |
+-------------------------------------------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| /|\
| Q A X F A B R I C I N T E R F A C E |
\|/ |
RP/0/RP0/CPU0:ma0#show controllers npu diag pp ?
EncapsulationInfo Dump last packet encapsulation detail(cisco-support)
ForwardDecisionTrace Basic forwarding information for last ingress frame(cisco-support)
ParsingInfo Dump last packet detail from parser(cisco-support)
PktAssociatedTmInfo Dump last packet Forwarding id detail(cisco-support)
ReceivedPacketInfo Dump last recieved packet detail(cisco-support)
TerminationInfo List of programs used for termination(cisco-support)
TrapsInfo Last Trap information(cisco-support)
#show controllers fia statistics detail instance all location all
RP/0/RP0/CPU0:ma0#show controllers npu resources ?
all all hardware resources
ecmpfec ecmp_fec
encap encap
exttcamipv4 ext_tcam_ipv4
exttcamipv6 ext_tcam_ipv6
exttcamipv6long ext_tcam_ipv6_long
exttcamipv6short ext_tcam_ipv6_short
fec fec
lem lem
lpm lpm
stats Show stats hardware resources(cisco-support)
RP/0/RP0/CPU0:ma0#show controllers npu stats traps-all instance 0 location 0/0/CPU0
Tue Nov 2 10:16:28.203 GMT
Trap Type NPU Trap TrapStats Policer Packet Packet
ID ID ID Accepted Dropped
============================================================================================================================
RxTrapMimDiscardMacsaDrop (IRB) 0 1 0x1 32045 0 0
RxTrapMimDiscardMacsaTrap (ERP_BDL) 0 2 0x2 32041 0 0
RxTrapMimDiscardMacsaSnoop(dot1x) 0 3 0x3 32020 0 0
RxTrapMimTeSaMove(SPIO) 0 4 0x4 32041 6434471 0
RxTrapMimSaMove(CFM_DOWM_MEP_DMM) 0 6 0x6 32039 0 0
RxTrapMimSaUnknown(RCY_CFM_DOWN_MEP_DMM) 0 7 0x7 32039 0 0
RxTrapAuthSaLookupFail (IPMC default) 0 8 0x8 32035 0 0
RxTrapAuthSaPortFail (L3 wrong MAC) 0 9 0x9 32020 17 0
RxTrapAuthSaVlanFail (L3 unknown-MC/BC) 0 10 0xa 32020 95 5
RxTrapSaMulticast 0 11 0xb 32020 1 0
bcmRxTrapTerminatedSaMulticast 0 12 0xc 32020 0 0
RxTrapArpMyIp 0 14 0xe 32001 0 0
RxTrapArp 0 15 0xf 32001 51 0
RxTrapDhcpv4Server 0 19 0x13 32019 0 0
RxTrapDhcpv4Client 0 20 0x14 32024 0 0
RxTrapDhcpv6Server 0 21 0x15 32019 0 0
RxTrapDhcpv6Client 0 22 0x16 32024 0 0
RxTrapL2Cache_LACP 0 24 0x18 32003 0 0
RxTrapL2Cache_LLDP1 0 25 0x19 32004 0 0
RxTrapL2Cache_LLDP2 0 26 0x1a 32004 982956 0
RxTrapL2Cache_ELMI 0 28 0x1c 32005 0 0
RxTrapL2Cache_BPDU 0 29 0x1d 32029 0 0
RxTrapL2Cache_BUNDLE_BPDU 0 30 0x1e 32029 0 0
RxTrapL2Cache_CDP 0 31 0x1f 32002 12 0
RxTrapHeaderSizeErr 0 33 0x21 32020 0 0
bcmRxTrapTerminatedHeaderSizeErr 0 34 0x22 32020 0 0
RxTrapIpCompMcInvalidIp 0 37 0x25 32020 0 0
RxTrapMyMacAndIpDisabled 0 38 0x26 32020 7 1
RxTrapMyMacAndMplsDisable 0 39 0x27 32020 0 0
RxTrapArpReply 0 40 0x28 32001 1364 0
RxTrapFibDrop 0 43 0x2b 32020 0 0
RxTrapMTU 0 44 0x2c 32022 0 0
RxTrapMiscDrop 0 45 0x2d 32020 0 0
RxTrapL2AclDeny 0 46 0x2e 32036 0 0
Rx_UNKNOWN_PACKET 0 48 0x30 32020 80882737 37765167
RxTrapL3AclDeny 0 49 0x31 32036 0 0
RxTrapStpStateBlock_EXT_HDR 0 51 0x33 32048 0 0
RxTrapFailover1Plus1Fail 0 55 0x37 32043 0 0
RxTrapOamY1731MplsTp(OAM_SWOFF_DN_CCM) 0 59 0x3b 32031 0 0
RxTrapOamY1731Pwe(OAM_SWOFF_UP_CCM) 0 60 0x3c 32032 0 0
RxTrapOamLevel 0 67 0x43 32025 0 0
RxTrapRedirectToCpuOamPacket 0 68 0x44 32027 0 0
RxTrapOamPassive 0 69 0x45 32026 0 0
RxTrap1588 0 70 0x46 32040 0 0
RxTrapExternalLookupError 0 71 0x47 32020 0 0
RxTrapArplookupFail 0 72 0x48 32001 0 0
RxTrapL2cpDrop (INVALID L2CP) 0 74 0x4a 32046 0 0
RxTrapTrillUnknownUc (flooding UC disable) 0 81 0x51 32044 0 0
RxTrapTrillUnknownMc (flooding MC-BC disable) 0 82 0x52 32044 0 0
RxTrapMplsControlWordTrap 0 86 0x56 32017 0 0
bcmRxTrapTerminatedMplsControlWordTrap 0 87 0x57 32017 0 0
RxTrapMplsControlWordDrop 0 88 0x58 32017 0 0
RxTrapMplsUnknownLabel 0 90 0x5a 32020 57293188 6651677
RxTrapIpv4VersionError 0 98 0x62 32020 8 0
bcmRxTrapTerminatedIpv4VersionError 0 99 0x63 32020 0 0
RxTrapIpv4ChecksumError 0 100 0x64 32020 0 0
bcmRxTrapTerminatedIpv4ChecksumError 0 101 0x65 32020 0 0
RxTrapIpv4HeaderLengthError 0 102 0x66 32020 0 0
bcmRxTrapTerminatedIpv4HeaderLengthError 0 103 0x67 32020 0 0
RxTrapIpv4TotalLengthError 0 104 0x68 32020 0 0
bcmRxTrapTerminatedIpv4TotalLengthError 0 105 0x69 32020 0 0
RxTrapIpv4Ttl0 0 106 0x6a 32010 0 0
bcmRxTrapTerminatedIpv4Ttl0 0 107 0x6b 32010 0 0
RxTrapIpv4Ttl1 0 110 0x6e 32010 317670 0
bcmRxTrapTerminatedIpv4Ttl1 0 111 0x6f 32010 0 0
RxTrapIpv4DipZero 0 113 0x71 32020 0 0
bcmRxTrapTerminatedIpv4DipZero 0 114 0x72 32020 0 0
RxTrapIpv4SipIsMc 0 115 0x73 32020 0 0
bcmRxTrapTerminatedIpv4SipIsMc 0 116 0x74 32020 0 0
RxTrapIpv6VersionError 0 118 0x76 32020 0 0
bcmRxTrapTerminatedIpv6VersionError 0 119 0x77 32020 0 0
RxTrapIpv6HopCount0 0 120 0x78 32013 0 0
RxTrapIpv6LoopbackAddress 0 123 0x7b 32020 0 0
bcmRxTrapTerminatedIpv6LoopbackAddress 0 124 0x7c 32020 0 0
RxTrapIpv6MulticastSource 0 125 0x7d 32020 0 0
bcmRxTrapTerminatedIpv6MulticastSource 0 126 0x7e 32020 0 0
RxTrapIpv6NextHeaderNull 0 127 0x7f 32012 0 0
bcmRxTrapTerminatedIpv6NextHeaderNull 0 128 0x80 32012 0 0
RxTrapSRv6Cnt3 0 131 0x83 32020 0 0
bcmRxTrapTerminatedIpv6LocalSiteDestination 0 132 0x84 32020 0 0
RxTrapIpv6Ipv4CompatibleDestination 0 135 0x87 32020 0 0
bcmRxTrapTerminatedIpv6Ipv4CompatibleDestination 0 136 0x88 32020 0 0
RxTrapMplsTtl0 0 139 0x8b 32014 0 0
RxTrapMplsTtl1 0 140 0x8c 32014 91 0
RxTrapMplsTtl1Terminated 0 141 0x8d 32014 0 0
RxTrapUcStrictRpfFail 0 153 0x99 32037 0 0
bcmRxTrapUcLooseRpfFail 0 154 0x9a 32037 0 0
RxTrapMcExplicitRpfFail 0 155 0x9b 32035 0 0
RxTrapOamp(OAM_BDL_DN_NON_CCM) 0 158 0x9e 32033 0 0
RxTrapOamEthUpAccelerated(OAM_BDL_UP_NON_CCM) 0 162 0xa2 32034 0 0
RxTrapReceive 0 167 0xa7 32019 272344642 0
RxTrapUserDefine_FIB_IPV4_NULL0 0 168 0xa8 32020 0 0
RxTrapUserDefine_SR_Unknown_Label 0 169 0xa9 32020 43 315
RxTrapUserDefine_FIB_IPV6_NULL0 0 170 0xaa 32020 0 0
RxTrapUserDefine_FIB_IPV4_GLEAN 0 171 0xab 32018 83 7791
RxTrapUserDefine_FIB_IPV6_GLEAN 0 172 0xac 32018 0 0
RxTrapUserDefine_IPV4_OPTIONS 0 173 0xad 32006 0 0
RxTrapUserDefine_IPV4_RSVP_OPTIONS 0 174 0xae 32007 0 0
RxTrapUserDefine 0 175 0xaf 32028 0 0
RxTrapUserDefine_IPV4_PIM_OPTIONS 0 177 0xb1 32008 0 0
RxTrapUserDefine_RECEIVE_L2 0 178 0xb2 32019 2665571 0
RxNetflowSnoopTrap0 0 202 0xca 32020 0 0
RxNetflowSnoopTrap1 0 203 0xcb 32020 0 0
bcmRxTrapUnknownDest 0 207 0xcf 32020 0 0
RxTrapSrv6EndOpPunt 0 208 0xd0 32047 0 0
bcmRxTrapMacSecurity 0 209 0xd1 32049 0 0
RxTrapUserDefine_OAM_SAT 0 212 0xd4 32050 0 0
# Some of the counters in "show controllers npu stats counters-all detail" have ambiguous names.
# Example below:
RP/0/RP0/CPU0:ma0# show controllers npu stats counters-all detail instance 3 location 0/3/CPU0 | inc "Drop|Rjct|Dscrd|Err|Del" | ex ": 0"
Fri Nov 12 11:22:53.514 GMT
IQM0 IqmCntCmdErrorsFilterA : 255
IQM0 IqmCntCmdErrorsFilterB : 255
IQM0 IrppCntCmdErrorsFilterA : 255
IQM0 IrppCntCmdErrorsFilterB : 255
IQM1 IqmCntCmdErrorsFilterA : 255
IQM1 IqmCntCmdErrorsFilterB : 255
IQM1 IrppCntCmdErrorsFilterA : 255
IQM1 IrppCntCmdErrorsFilterB : 255
NBIH BistRxErrLengthPktCnt : 15528
NBIH BistRxErrPktIndexCnt : 59950
NBIH BistRxErrDataCnt : 11002
NBIH BistRxErrInCrcErrCnt : 56425
NBIH BistRxErrSobCnt : 34699
Regarding BIST counters:
BistRxErrPktIndexCnt
BistRxErrDataCnt
BistRxErrInCrcErrCnt
BistRxErrSobCnt
"These error counters are harmless and can be ignored. These are caused by NBI BIST. Nbi Bist function is intended as a traffic generator & checker when the external port is looped back. But, unfortunately, the checker is running all the time, even when the traffic generator is not being used. The only time the counts are reliable is if the port is in external loopback, and traffic is coming from the NBI BIST traffic generator, rest of the time those numbers are invalid. These cannot be disabled too."
StEgrBillDeqRprtDropCtr
StIngBillDeqRprtDropCtr
StIngBillEnqRprtDropCtr
"All these 3 drop counters above are harmless ..they can be ignored."
ErppLagPruningDscrdsCnt
"This counter is expected in cases where lag has more than 1 members and multicast traffic needs replication on lag port. In that case, only one lag member will be chosen and rest of the members will drop the packet due to lag pruning. in such case, above counter will increment."
EhpDscrdPktCnt
EpeDscrdPktsCnt
"These are EGQ drops (inside the Egress Traffic Manager block). They may be related to egress congestion ..you should check “diag egr_congestion” output multiple times when these drops are incrementing."
# Remember that with the following command, the packet/byte counts are for traffic leaving the named interface (Te0/0/0/24).
# The command is showing traffic that ingressed an VoQ which will egress via the named interfaces. So despite saying "ingress"
# this command is showing egress traffic:
#
RP/0/RP0/CPU0:ma0#show controllers npu stats voq ingress interface Te0/0/0/24 instance 0 location 0/0/CPU0
Tue Nov 2 10:24:47.723 GMT
Interface Name = Te0/0/0/24
Interface Handle = 48
Asic Instance = 0
VOQ Base = 1216
Port Speed(kbps) = 10000000
Local Port = local
ReceivedPkts ReceivedBytes DroppedPkts DroppedBytes
-------------------------------------------------------------------
TC_0 = 15 1500 0 0
TC_1 = 0 0 0 0
TC_2 = 0 0 0 0
TC_3 = 218765888 376013039231 0 0
TC_4 = 0 0 0 0
TC_5 = 0 0 0 0
TC_6 = 0 0 0 0
TC_7 = 0 0 0 0
Previous page: NCS540 Packet Capture
Next page: FIA Packet Diag