ME3600/ME3800 - Hardware Troubleshooting

Troubleshooting Commands:

BFD drops:

show bfd drops
show bfd neighbor detail
show processes cpu sorted
show platform qos policer cpu  1 0      !  Check CPU policer for BFD packets
show platform aspdma all_counters 0
sh platform ho-fpga tx-buffer-table de 1    !  ME3600-CX only
show platform ho-fpga tx-buffer-table detail 258  !  ME3600-CX only

 

CPU usage & punts:

show controllers cpu-interface

 

Interface and ASIC:

show controllers utilization
show platform asic ecc 0
show platform asic ecc 1
show platform asic stats 0
show platform asic stats 1
show platform aspdma all_counters 0
show platform aspdma all_counters 1
! Drill down into the above output per ASIC queue using
show platform aspdma rx_stats <0-1>  rx device id <0-31>  rx queue number
! and..
show platform aspdma tx_stats <0-1>  tx device id <0-3>  tx queue number
test platform pm force-link { Gi | Te } X/X  { Status | Enable | Disable}
test platform pm phy { Gi | Te } X/X { up | down }

sdcli
  nile stats ipstats 0  ! ASIC Statis
  nile stats ipstats 1
  nile stats channel 0  ! ASIC Statis
  nile stats channel 1
  nile stats zero_stats 0 0 all  ! Zero all ASIC stats
  nile stats zero_stats 1 0 all
  nile debug stats { 0 | 1 } ForwardProcessorStats
  nile debug stats { 0 | 1 } ?  ! Many options here

It possible to tickle the ASIC registers to dump packets passing through them to the console, however, even in the lab this produced so much output the console became unusable:

show platform aspdma rx_q_pkts <0-31>  cpu queue <0-1>  1=enable 0=disable
show platform aspdma tx_asic <0-1>  Asic number <0-3>  Queue number  <0-1>  1=mcast, 0=ucast  <0-1>  1=enable, 0=disable

! Example showing an LDP "Hello" packet arriving on Rx queue 26:

ME3600#show platform aspdma rx_q_pkts 26 1
*Oct  2  16:52:37.273 BST: NH[0]  = A8040234
*Oct  2  16:52:37.273 BST: NH[1]  = 0000037A
*Oct  2  16:52:37.273 BST: NH[2]  = 00000000
*Oct  2  16:52:37.273 BST: NH[3]  = 00F69500
*Oct  2  16:52:37.273 BST: NH[4]  = 00581F56
*Oct  2  16:52:37.273 BST: NH[5]  = 07160001
*Oct  2  16:52:37.273 BST: NH[6]  = 0010001A
*Oct  2  16:52:37.273 BST: NH[7]  = 00000000
*Oct  2  16:52:37.273 BST: | Mcast: 0 LenDelta 0    DestIndx 0x0000037A |
*Oct  2  16:52:37.273 BST: | VE: 0    CVlanId 0    Priority 1    FlowHash 7    Vlan 1    |
*Oct  2  16:52:37.273 BST: | L2IntID: 0x00000058 L3IntId: 0x00001F56 AdjPtr: 0x0000001A |
*Oct  2  16:52:37.273 BST: | PhyId: 22   PayloadOffset 0    Type 1    |
*Oct  2  16:52:37.273 BST: pkt size: 76 bytes_to_print: 1514
*Oct  2  16:52:37.273 BST: byte:0   01  00  5E  00  00  02  1C  DE
*Oct  2  16:52:37.273 BST: byte:8   A7  9A  19  D7  08  00  45  C0
*Oct  2  16:52:37.273 BST: byte:16  00  3E  00  00  00  00 01  11
*Oct  2  16:52:37.273 BST: byte:24  D6  E9  02  02  00  02  E0  00
*Oct  2  16:52:37.273 BST: byte:32  00  02  02  86  02  86  00  2A
*Oct  2  16:52:37.273 BST: byte:40  0B  37  00  01  00  1E  01  01
*Oct  2  16:52:37.273 BST: byte:48  01  02  00  00  01  00  00  14
*Oct  2  16:52:37.273 BST: byte:56  00  00  00  00  04  00  00  04
*Oct  2  16:52:37.273 BST: byte:64  00  0F  00  00  04  01  00  04
*Oct  2  16:52:37.273 BST: byte:72  01  01  01  02

 

TCAM partitioning and usage:

show licence
show platform aspdma template
! MPLS label resource limits
show platform aspdma template | include MPLS
show platform tcam usage
show platform tcam utilization all
show platform nile adjmgr all
! MPLS label usage
show platform nile adjmgr all | include EMPLS
show platform template
show sdm prefer current
sdm prefer { default | ip | vpnv4-only | vpnv4-v6 }

 

VLAN issues:

show platform qos vlan-list     ! Show VLAN to port mappings

 

QoS issues:

conf t; service internal;
show platform tcam utilization qos ! TCAM utilisation for QoS lookups
show platform qos scheduler profile queue 0 ! Scheduler profiles, max 255, scheduler is a different hardware resource, and TCAM utilization is independent of this.
show platform qos tree ethernet service-instance id 2279 interface gigabitEthernet 0/1 output|output all
show platform qos default-eqos ethernet service-instance id 2279 interface gi0/1 stats
show platform qos scheduler resource-usage

---