6500/7600 NetDR Captures

References:
https://supportforums.cisco.com/document/59956/troubleshooting-netdr-capture-sup7206500
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/116475-technote-product-00.html
https://supportforums.cisco.com/document/53866/troubleshooting-high-cpu-under-interrupts-7600-and-6500-boxes-using-debug-netdr-tool
http://netdr.54.227.241.219.xip.io
http://www.cisco.com/c/en/us/td/docs/routers/7600/ios/15S/configuration/guide/7600_15_0s_book/dos.html#pgfId-1163901
https://cway.cisco.com/tools/netdr/

NetDR can capture traffic coming into/out of the MSFC on the 6500/7600 Sup/RSP. It is production safe. It captures 4096 packets by default (which can't be lowered) The capture can be stopped with "undebug netdr capture".

conf t

 ip access-list extended 199
  remark NetDR ACL - Exclude this SSH session
  10 deny tcp host 192.168.30.122 any eq 22
  20 deny tcp any eq 22 host 192.168.30.122
  end

7600#debug netdr capture rx acl 199

7600#show netdr captured-packets
A total of 4058 packets have been captured
The capture buffer wrapped 0 times
Total capture capacity: 4096 packets

------- dump of incoming inband packet -------
interface NULL, routine process_rx_packet_inline, timestamp 10:18:32.579
dbus info: src_vlan 0x44C(1100), src_indx 0xC0(192), len 0x70(112)
  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
  48020C00 044C0000 00C00000 70000000 00110418 2E000008 00000000 038039C7 
destmac C4.71.FE.02.5E.00, srcmac 00.00.00.00.00.00, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 94, identifier 14503
  df 0, mf 0, fo 0, ttl 127, src 192.168.30.33, dst 10.254.254.1
    udp src 57966, dst 161 len 74 checksum 0x7E54

The interface LTL index can be looked-up to see where traffic is coming from / going to. In this case where the destination index in the RP, we can look up that which registers in the ASIC have that interface index as their value:

7600#remote command switch test mcast ltl-info index C0

index 0xC0 contain ports 4/1

7600#remote command switch test mcast ltl-info index 380

index 0x380 contain ports 5/R


7600#remote command switch show platform hardware tycho register 0 1794 | i 0x00000380
 0x017F:            PP_RF_SRC_IDX0 = 0x00000380 [896       ]
 0x03C4:            RED_SW_ERR_IDX = 0x00000380 [896       ]
 0x0456:            RED_FINRST_IDX = 0x00000380 [896       ]
 0x045B:          RED_IPV6_SCP_IDX = 0x00000380 [896       ]

Traffic levels into the RP can be checked using "show ibc":

7600#show ibc
Interface information:
        Interface IBC0/0(idb 0x1CA02D48)
        5 minute rx rate 72000 bits/sec, 101 packets/sec
        5 minute tx rate 102000 bits/sec, 86 packets/sec
        691696374 packets input, 71610233739 bytes
        141407749 broadcasts received
        589386626 packets output, 146461004523 bytes
        220161291 broadcasts sent
        0 Bridge Packet loopback drops
        154846506 Packets CEF Switched, 12485 Packets Fast Switched
        0 Packets SLB Switched, 0 Packets CWAN Switched
        Label switched pkts dropped: 689    Pkts dropped during dma: 1200
        Invalid pkts dropped: 0    Pkts dropped(not cwan consumed): 0
        Pkts marked to drop by VLAN clients: 0
        IPSEC pkts: 15375
        Xconnect pkts processed: 29007, dropped: 0
        Xconnect pkt reflection drops: 0
        Total paks copied for process level 29007
        Total short paks sent in route cache 49117602
        Total throttle drops 1149    Input queue drops 26
        total spd packets classified (237191924 low, 219915635 medium, 45881993 high)
        total spd packets dropped (1101 low, 97 medium, 2 high)
        spd prio pkts allowed in due to selective throttling (0 med, 0 high)
        IBC resets   = 1; last at 11:03:12.551 BST Thu Oct 15 2015

Driver Level Counters: (Cumulative, Zeroed only at Reset)
          Frames          Bytes
  Rx(0)   49851708        1746625990
  Rx(1)   641845991       4283499493
  Tx(0)   589386823       4186294699

 Input Drop Frame Count
     Rx0 = 0                Rx-replacement0 = 0                Rx1 = 0                Rx-replacement1 = 0
 Per Queue Receive Errors:
     FRME   OFLW   BUFE   NOENP  DISCRD DISABLE BADCOUNT
 Rx0 0      0      0      0      0        0    0
 Rx1 0      0      0      0      0        0    0


  Tx Errors/State:
   One Collision Error   = 0            More Collisions       = 0
   No Encap Error        = 0            Deferred Error        = 0
   Loss Carrier Error    = 0            Late Collision Error  = 0
   Excessive Collisions  = 0            Buffer Error          = 0
   Tx Freeze Count       = 0            Tx Intrpt Serv timeout= 1
   Tx Flow State         = FLOW_ON
   Tx Flow Off Count     = 0            Tx Flow On Count      = 0

  Counters collected at Idb:
   Is input throttled    = 0            Throttle Count        = 0
   Rx Resource Errors    = 0            Input Drops           = 1
   Input Errors           = 6
   Output Drops          = 0            Giants/Runts          = 0/0
   Dma Mem Error         = 0            Input Overrun         = 0



abr1#show platform netint
 Interrupt Throttling:
  Throttle Count   = 00011967   Timer Count      = 00011965
  Netint usec      = 00001000   Netint Mask usec = 00002000
  Active           =        0   Configured       =        1
  Longest IRQ(usec)= 00001738




abr1#show mls statistics module 6

Statistics for Earl in Module 6

L2 Forwarding Engine
  Total packets Switched                : 2208565183727

L3 Forwarding Engine
  Total packets Processed               : 2208480580634 @ 1101627 pps
  Total packets L3 Switched             : 1977298775705 @ 1001275 pps

  Total Packets Bridged                 : 21442811858
  Total Packets FIB Switched            : 1977298775705
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 0
  Total Mcast Packets Switched/Routed   : 172529619
  Total ip packets with TOS changed     : 28856210277
  Total ip packets with COS changed     : 33170721579
  Total non ip packets COS changed      : 74005042008
  Total packets dropped by ACL          : 16572339
  Total packets dropped by Policing     : 126705455
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 0
  Short IP packets received             : 0
  IP header checksum errors             : 0
  No-route packet drops                 : 991546492
  TTL failures                          : 23957092
  MTU failures                          : 655181

---