PFE Commands

Egress Lookup

JSIM is like 7600 ELAM, we can build a parcel and send it to the PFE/PPE and check the response.

NPC0(abr2-ld5slo.core vty)# set jsim input-port wan
NPC0(abr2-ld5slo.core vty)# set jsim ipsrc 10.0.0.50
NPC0(abr2-ld5slo.core vty)# set jsim ipdst 10.0.0.20
NPC0(abr2-ld5slo.core vty)# set jsim ip-protocol 1
NPC0(abr2-ld5slo.core vty)# show jsim packet
0000000000000000
0000000000000000
0000080045000032
0001000020018685
0a0000320a000014
0000000000000000
0000000000000000
0000000000000000
000000000000

NPC0(abr2-ld5slo.core vty)# jsim run 1
[May 11 12:22:10.114 LOG: Info] TTRACE PPE14 Context2 now in-active.

 

Use "show jnh lb" to look at load-balancing.

 

Filters

Show filters (such as CoPPs) programmed in TCAM:

start shell pfe network fpc0
show filter
show filter hw <lo0 filter from above> show_term_info
# Check the number of required TCAM entries doesn't exceed the number available
show filter hw <from above> show_terms_brcm # This may crash PFE if TCAM is incorrectly programmed
# Some devices have dynamic filters built-in like QFX
show filter hw dynamic 47 show_terms

 

Next Hop

Show the next-hop entry:

> start shell pfe network fpc0
> show nhdb id 1307 detail

 

Packet-via-dmem

start shell pfe network fpc<number>

# May need to elevate permission in PFE
set parser security 10


# Disabe disable liveness, fabric self-ping, and all callouts, to stop them appearing in the capture buffer
set host_loopback disable-periodic
test fabric self_ping disable <pfe-number>
test jnh <pfe-instance> pfe-liveness timeout 0
test jnh <pfe-instance> pfe-liveness stop


test jnh <pfe-instance> packet-via-dmem enable [optional buffer size here - default is 16k]
# Every parcel received by LU processing and sent to MQ/XM at the end of LU processing will be saved

test jnh <pfe-instance> packet-via-dmem capture <capture type> <capture mask>

# The capture type is the type of packet/parcel to capture:
#define M2L_PKT 0               ---> 1 << 0 = 1
#define M2L_PKT_HEAD 1          ---> 1 << 1 = 2 1 | 2 = 0x0003
#define M2L_BACKPRESSURE 2
#define M2L_PKT_DATA_RD_REPLY 3
#define M2L_PKT_DATA_WR_REPLY 4
#define M2L_PARCEL_TYPE05 5
#define M2L_PARCEL_TYPE06 6
#define M2L_PARCEL_TYPE07 7
#define M2L_STATS_UPDATE 8
#define M2L_GAUGE 9
#define M2L_PARCEL_TYPE10 10
#define M2L_PARCEL_TYPE11 11
#define M2L_PARCEL_TYPE12 12
#define M2L_PARCEL_TYPE13 13
#define M2L_TIMER_WCLOCK_CALLOUT 14
#define M2L_ERROR 15

# The mask is simply hex bytes that appear anywhere in the packet.
# AABBCC could be part of the SRC MAC or DEST MAC or THE IP ADDR, whatever.
# This is also optional.

test jnh <pfe-instance> packet-via-dmem capture 0x03 AABBCC
test jnh <pfe-instance> packet-via-dmem capture 0x10003

# If multiple LU chips are present, specify the LU chip after the PFE.
# If none is specified capture is run on all LUs - each with a 16k buffer (assuming the default buffer wasn't changed)

test jnh <pfe-instance> <lu-instance> packet-via-dmem capture 0x3

# Wait for capture
test jnh <pfe-instance> packet-via-dmem capture 0 # 0 means stop
test jnh <pfe-instance> packet-via-dmem dump
# Dump from specific LU when dump was run on all LUs on multi-LE PFE
test jnh <pfe-instance> <lu-instance> packet-via-dmem dump

# When finished disable capture
test jnh <pfe-instance> packet-via-dmem disable

 

SFP

Show SFP speed:

> show interfaces xe-5/0/9
Physical interface: xe-5/0/9, Enabled, Physical link is Up
  Interface index: 174, SNMP ifIndex: 710
  Link-level type: Ethernet, MTU: 9300, MRU: 9200, LAN-PHY mode, Speed: 10Gbps

> start shell pfe network fpc0

# MM fibre example:

>show sfp list

Index   Name
-----   --------------
    1     CMIC(5/0)(9)

> show sfp 1

...
  Wavelength           850 nm
...
   Fiber mode           MM



# Copper example:

>show sfp list

Index   Name             Presence     ID Eprom  PNO         SNO          calibr
-----   --------------   ----------   --------  ----------  ----------- -------
    1     XETH(2/0)(0)       Absent   Partial
    2     XETH(2/0)(1)      Present   Complete  NON-JNPR  XXXXXXXX

> show sfp 2

...
   SFP xcvr codes:      10/100/1000Base-T SFP
...
   SFP Type:            1000BASE-T Copper SFP

 

Troubleshooting

start shell pfe network <fpc#>
show syslog messages

show route summary
show jtree 0 summary
show jtree 0 memory

show jnh 0 pool summary
show jnh 0 pool bytes
show jnh 0 pool usage
show jnh 0 pool stats <number> # (0-NH, 1-DFW, 2-CNT, 7-UEID, 8-SHARED_UEID)

show hsl2 statistics
show precl-eng summary
show precl-eng 1 statistics

show luchip 0
show luchip 0 errmsg

show trinity_pio errors

show mqchip 0 ifd
show xmchip 0 ifd list 0 # second 0 means Ingress – 1 Egress
show xr2chip

 

ttrace

Problem:
On Trinity-based platform, PFE wedge and FPC restart caused by TTRACE failure might happen in rare cases. Log messages as follows might also be observed.

%PFE-3: fpc7 LUCHIP(1): Secondary PPE 0 zone 1 timeout.
%PFE-3: fpc7 PPE Thread Timeout Trap: Count 864, PC 68b, 0x068b: KT4Way_BTT0_ARS00 0x068b: KT4Way_BTT0_ARS10
%PFE-5: fpc7 TTRACE PPE4 Context12 buffer full.
%PFE-3: fpc7 Host Loopback:HOST LOOPBACK WEDGE DETECTED IN PATH ID 3
%PFE-3: fpc7 TOE Pkt Xfer:** WEDGE DETECTED IN PFE 1 stream 4 TOE host packet transfer: reason code 0x1

Triggers:
This issue might be seen if the following conditions are met:
* On Trinity based PFE (e.g. MPC on MX series, FPC5 on T4000 router)
* Auto-ttrace feature enabled (which is default)
* Multicast replication with a large amount of copies or with additional features like fragmentation or output firewall filters

PR1139406 - Trinity PFE wedge and FPC restart with large-scale multicast replication

Auto-ttrace feature is enabled by default on FPCs with a trinity PFE. PPE (Packet Processing Engine) traps cause auto-ttrace to capture detailed information of packets for future debugging. In some cases, that can keep the LU thread busy too long and eventually may lead to a wedge of the LU/XM or XL/XM PFE complex and a restart of the respective FPC. This can happen if the forwarding-lookup involves multicast replication with a large amount of copies or multicast replication with additional features like fragmentation or output firewall filters.

Resolved In 13.3R10 14.1R8 14.2R6 15.1R3 15.1F2-S17 15.1F5-S2 15.1F6 16.1R1

Workaround:

Disable the auto-ttrace as below.

# Connecting XM based FPC's shell
start shell pfe network fpc(number) 

# Setting Security level to 15
set parser security 15

# Checking the current status for Auto-ttrace
show jnh 0 trap-info struct

#Disabling the auto-ttracetest jnh 0 trap-info-read trace disable 

# Checking the change for Auto-ttrace
show jnh 0 trap-info struct


NPC3(poplar-re0 vty)# set parser security 15

NPC3(poplar-re0 vty)# show jnh 0 trap-info struct 
TRAP INFO Config:
trap_rd_sem: 0x4ca5f618
trap_rd_thread: 0x46d1e608
flags: SAVE TRACE <<<<<<<<<<<<<<<<
save_count: 32
trap_info_savefile: /var/tmp/ppe_trap_fpc3_LU_0_00

NPC3(poplar-re0 vty)# test jnh 0 trap-info-read trace disable

NPC3(poplar-re0 vty)# show jnh 0 trap-info struct 
TRAP INFO Config:
trap_rd_sem: 0x4ca5f618
trap_rd_thread: 0x46d1e608
flags: SAVE <<<<<<<<<<<<<<<<
save_count: 32
trap_info_savefile: /var/tmp/ppe_trap_fpc3_LU_0_00

 

---