Date created: Tuesday, October 15, 2024 11:38:10 AM. Last modified: Tuesday, April 15, 2025 5:41:40 PM
7280R3 - Finding Packet Loss
References
https://arista.my.site.com/AristaCommunity/s/article/platform-specific-discards
https://arista.my.site.com/AristaCommunity/s/article/troubleshooting-egress-queue-drops-on-7280-7500-devices
https://www.arista.com/en/support/toi/eos-4-26-0f/14716-support-for-differentiated-mmu-discard-counters-on-7060x4-and-7368x4-series
https://www.arista.com/en/support/toi/eos-4-27-0f/14843-drop-precedence-subinterface
https://arista.my.site.com/AristaCommunity/s/question/0D52I00007Ja85tSAB/questions-about-maxquesizemaxbufferquesize-with-voq
https://arista.my.site.com/AristaCommunity/s/article/What-is-a-traffic-black-holing-effect
show interface
lab1#show interfaces Eth50/1
Ethernet50/1 is up, line protocol is up (connected)
Hardware is Ethernet, address is ac3d.94aa.bbcc (bia ac3d.94aa.bbcc)
Ethernet MTU 10218 bytes, Ethernet MRU 10240 bytes, BW 100000000 kbit
Full-duplex, 100Gb/s, auto negotiation: off, uni-link: disabled
Up 11 days, 18 hours, 1 minute, 10 seconds
Loopback Mode : None
0 link status changes since last clear
Last clearing of "show interface" counters 11 days, 17:49:23 ago
5 seconds input rate 658 Mbps (0.7% with framing overhead), 107093 packets/sec
5 seconds output rate 1.63 Gbps (1.7% with framing overhead), 186043 packets/sec
303476033196 packets input, 187544478586224 bytes
Received 20171 broadcasts, 726115 multicast
8863 runts, 0 giants
151338676 input errors, 144801071 CRC, 0 alignment, 6528742 symbol, 17040448 input discards
0 PAUSE input
586132914713 packets output, 665804887152081 bytes
Sent 20175 broadcasts, 726636 multicast
0 output errors, 0 collisions
0 late collision, 0 deferred, 0 output discards
0 PAUSE output
The output of "show interface X" includes the following counters:
- Input runts, giants, CRC, alignment, and symbol counters, are counters for each type of error that can occur for a frame received over the link.
- Input errors in the total count of all of these individual counters.Input discards are discards during ingress queuing and buffering (i.e. buffer overflow). VoQ is used to avoid head-of-line blocking. Each ingress interface has a VoQ per traffic class per egress interface. Traffic is queued in the ingress interface until a credit is issued by the ASIC or fabric. Then the packet is sent towards the egress interface and transmitted. Input discards happen because the VoQ for a specific traffic class of a specific egress interface is full, and more traffic for that VoQ has been received.
- PAUSE input counts the number of Ethernet PAUSE frames received when flow control is enabled.
- Collisions, late collisions, and deferred are individual counters for each type of error that can occur when transmitting a frame.
- Output errors is the total across all of these individual counters.
- Output discards show packet drops from the interface egress buffer. Even though VoQs are used at the ingress interface, VoQs are only used for unicast traffic. This is because fabric/egress replication mode is used by default for BUM traffic (the credit mechanism for egress VoQs is not used). In this mode, BUM traffic is sent directly to the egress interface and thus the egress buffer can overflow during periods of high volumes of BUM traffic. One reason for this is so that latency sensitive multicast traffic isn’t delayed. Replication of BUM traffic happens at the egress interface. On some platforms (7280R) the behaviour can be changed so that BUM traffic is replicated at ingress, and a packet is put into the VoQ of each destination interface and traffic class, so that the traffic is queued like unicast traffic. In the output of “show interface X”, the “output discards” counter shows the total discards across all egress VoQs. Individual VoQ drops per traffic class can be seen with “show interfaces x counters queue [drops]” under the section "Aggregate VoQ Counters", BUM traffic drops specifically can be seen under the section "Egress Queue Counters".
Note: I think the "output discards" also include "input discards" from an ingress interface that was trying to transmit to this interface and failed. - PAUSE output counts the number of Ethernet PAUSE frames sent when flow control is enabled.
The output of the following commands are included of the output of "show interface X" and don't need further exploring:
- show interfaces counters discards
- show interfaces counters errors
- show interfaces counters incoming
- show interfaces counters outgoing
The output of the following commands are not show in the output of "show interface x":
- show interfaces counters drop-precedence -> Not supported on 7280R
- show interfaces counters egress acl drop
- show interfaces counters egress mtu
- show interfaces counters half-duplex
- show interfaces counters ingress acl drop
- show interfaces counters queue
- show interfaces counters queue drops
- show interfaces counters queue ingress (the same info is shown in “show interfaces counters queue” under the “Aggregate VoQ Counters” section)
- show interfaces counters queue rates
show interfaces counters drop-precedence
lab#show interfaces counters drop-precedence
! Counters not supported on any interface
This requires drop-precedence counters to be enabled. Something like the following:
#hardware counter subinterface in drop-precedence
#hardware counter subinterface out drop-precedence
Although, these don't work on 7280R platform. Once enabled (on a supported platform) one has counters for how many packets were dropped based on the drop-precedence assigned to the packet in a policy map (the policy map is applied to a QoS profile, the QoS profile is configured on the interface).
show interfaces counters egress acl drop
Packets dropped by an egress ACL:
lab#show interfaces counters egress acl drop
Aggregate Egress Acl Counters
Port Source Pruning Mlag
Et1/1 n/a n/a
Et2/1 n/a n/a
Et3/1 n/a n/a
Et4/1 n/a n/a
show interfaces counters egress mtu
Packets dropped by an ingress/egress MTU mismatch:
lab#show interfaces counters egress mtu
Aggregate Egress Mtu Counters
Switch EnqueuePkts Discards TotalPkts
Fap0 n/a n/a n/a
show interfaces counters half-duplex
Packets dropped due to a half-duplex timing issue:
lab#show interfaces counters half-duplex
Single Multiple Late Excessive Deferred
Interface Collision Collision Collision Collision Transmission
--------- --------- --------- --------- --------- ------------
Et1/1 0 0 0 0 0
Et2/1 0 0 0 0 0
Et3/1 0 0 0 0 0
Et4/1 0 0 0 0 0
show interfaces counters ingress acl drop
Packets dropped by an ingress ACL:
lab#show interfaces counters ingress acl drop
Port InAclDrops
------------ ----------
Et1/1 0
Et2/1 0
Et3/1 0
Et4/1 0
Et5/1 0
show interfaces Eth1/1 counters queue
Under the “Aggregate VoQ Counters” section: Shows the total number number of packets and octets which were forwarded or dropped in each egress VoQ on a port (showing combined values for unicast and multicast traffic at each traffic class), the total of these values is the “packets output” counter under “show interface X”.
Under the “Egress Queue Counters” section: Shows the number number of packets and octets which were forwarded or dropped in each egress VoQ on a port. There is a queue per traffic class per port for unicast traffic and per traffic class per port for multicast traffic. Note that unless ingress replication mode is explicitly enabled, multicast traffic with use a different VoQ to unicast traffic by default.
lab#show interfaces Et7/1 counters queue
Aggregate VoQ Counters
Egress Traffic Pkts Octets DropPkts DropOctets
Port Class
--------------------------------------------------------------------------------------------------------------
Et7/1 TC0 0 0 0 0
Et7/1 TC1 925790872050 101647320716541 8830667079832 962843799937772
Et7/1 TC2 0 0 0 0
Et7/1 TC3 0 0 0 0
Et7/1 TC4 0 0 0 0
Et7/1 TC5 0 0 0 0
Et7/1 TC6 5 320 0 0
Et7/1 TC7 264890 40038469 0 0
Egress Queue Counters
Port Traffic DropPrec DestType OutEnqPkts OutEnqOctets OutDropPkts OutDropOctets
Class
Et7/1 TC0 DP0-3 UC 0 0 0 0
Et7/1 TC0 DP0-3 MC 0 0 0 0
Et7/1 TC1 DP0-3 UC 925789882933 101647216929222 989117 103787319
Et7/1 TC1 DP0-3 MC 0 0 0 0
Et7/1 TC2 DP0-3 UC 0 0 0 0
Et7/1 TC2 DP0-3 MC 0 0 0 0
Et7/1 TC3 DP0-3 UC 0 0 0 0
Et7/1 TC3 DP0-3 MC 0 0 0 0
Et7/1 TC4 DP0-3 UC 0 0 0 0
Et7/1 TC4 DP0-3 MC 0 0 0 0
Et7/1 TC5 DP0-3 UC 0 0 0 0
Et7/1 TC5 DP0-3 MC 0 0 0 0
Et7/1 TC6 DP0-3 UC 5 320 0 0
Et7/1 TC6 DP0-3 MC 0 0 0 0
Et7/1 TC7 DP0-3 UC 264890 38184239 0 0
Et7/1 TC7 DP0-3 MC 0 0 0 0
show interfaces counters queue drops
This command shows that 925790872050 packets were enqueued into ingress chip buffer, but 8830667079832 where not enqueued at ingress (egress VoQ was probably full).
925789882933 packets were enqueued into egress buffer, 989117 were dropped (probably the egress buffer was full).
lab#show interfaces Et7/1 counters queue drops Port Class EnqPkts EnqOctets DropPkts DropOctets ----------------------------------------------------------------------------------------------- Et7/1 TC0 VOQ 0 0 0 0 - Fap0.0 0 0 0 0 - Fap0.1 0 0 0 0 Egress queue 0 0 0 0 - DP0-3 UC 0 0 0 0 - DP0-3 MC 0 0 0 0 Et7/1 TC1 VOQ 925790872050 101647320716541 8830667079832 962843799937772 - Fap0.0 1151345 200717683 0 0 - Fap0.1 925789720705 101647119998858 8830667079832 962843799937772 Egress queue 925789882933 101647216929222 989117 103787319 - DP0-3 UC 925789882933 101647216929222 989117 103787319 - DP0-3 MC 0 0 0 0 Et7/1 TC2 VOQ 0 0 0 0 - Fap0.0 0 0 0 0 - Fap0.1 0 0 0 0 Egress queue 0 0 0 0 - DP0-3 UC 0 0 0 0 - DP0-3 MC 0 0 0 0 Et7/1 TC3 VOQ 0 0 0 0 - Fap0.0 0 0 0 0 - Fap0.1 0 0 0 0 Egress queue 0 0 0 0 - DP0-3 UC 0 0 0 0 - DP0-3 MC 0 0 0 0 Et7/1 TC4 VOQ 0 0 0 0 - Fap0.0 0 0 0 0 - Fap0.1 0 0 0 0 Egress queue 0 0 0 0 - DP0-3 UC 0 0 0 0 - DP0-3 MC 0 0 0 0 Et7/1 TC5 VOQ 0 0 0 0 - Fap0.0 0 0 0 0 - Fap0.1 0 0 0 0 Egress queue 0 0 0 0 - DP0-3 UC 0 0 0 0 - DP0-3 MC 0 0 0 0 Et7/1 TC6 VOQ 5 320 0 0 - Fap0.0 5 320 0 0 - Fap0.1 0 0 0 0 Egress queue 5 320 0 0 - DP0-3 UC 5 320 0 0 - DP0-3 MC 0 0 0 0 Et7/1 TC7 VOQ 264890 40038469 0 0 - Fap0.0 264890 40038469 0 0 - Fap0.1 0 0 0 0 Egress queue 264890 38184239 0 0 - DP0-3 UC 264890 38184239 0 0 - DP0-3 MC 0 0 0 0
show interfaces Et7/1 counters queue ingress
Show packets being queued into VoQs at ingress:
lab#show interfaces Et7/1 counters queue ingress Fap0.0: Egress Traffic Pkts Octets DropPkts DropOctets Port Class -------------------------------------------------------------------------------------------------------------- Et7/1 TC0 0 0 0 0 Et7/1 TC1 1151345 200717683 0 0 Et7/1 TC2 0 0 0 0 Et7/1 TC3 0 0 0 0 Et7/1 TC4 0 0 0 0 Et7/1 TC5 0 0 0 0 Et7/1 TC6 5 320 0 0 Et7/1 TC7 264890 40038469 0 0 Fap0.1: Egress Traffic Pkts Octets DropPkts DropOctets Port Class -------------------------------------------------------------------------------------------------------------- Et7/1 TC0 0 0 0 0 Et7/1 TC1 925789720705 101647119998858 8830667079832 962843799937772 Et7/1 TC2 0 0 0 0 Et7/1 TC3 0 0 0 0 Et7/1 TC4 0 0 0 0 Et7/1 TC5 0 0 0 0 Et7/1 TC6 0 0 0 0 Et7/1 TC7 0 0 0 0
show interfaces Et1/1 counters queue rates
Show traffic rate per VoQ:
lab#show interfaces Et1/1 counters queue rates
Et1/1 - load interval 5.0s
Offered load (include drop) Config Drop rates Output rates
bps pps % of % of bps SP/WRR bps pps bps pps % of % of WRR %
link total link total
----- ------ ----- ------ ------- ----- ------ ----- ----- ------ ----- ----- ------ -----
UC0 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
UC1 89.9M 11.7k 0.9% 100.0% N/A SP 0.0 0.0 89.9M 11.7k 0.9% 100.0% N/A
UC2 467.7 0.4 0.0% 0.0% N/A SP 0.0 0.0 467.7 0.4 0.0% 0.0% N/A
UC3 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
UC4 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
UC5 26.3 0.0 0.0% 0.0% N/A SP 0.0 0.0 26.3 0.0 0.0% 0.0% N/A
UC6 811.2 0.8 0.0% 0.0% N/A SP 0.0 0.0 811.2 0.8 0.0% 0.0% N/A
UC7 5.5k 7.8 0.0% 0.0% N/A SP 0.0 0.0 5.5k 7.8 0.0% 0.0% N/A
MC0 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
MC1 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
MC2 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
MC3 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
MC4 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
MC5 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
MC6 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
MC7 0.0 0.0 0.0% 0.0% N/A SP 0.0 0.0 0.0 0.0 0.0% 0.0% N/A
Total 89.9M 11.7k 0.9% 100.0% N/A 0.0 0.0 89.9M 11.7k 0.9% 100.0% N/A
show platform fap voq scheduling
This command shows the rate of traffic being sent into the ASIC/backplane from each traffic class on each ingress interface (this is a 7280R3 pizza-box with two ASICs back-to-back, hence Fap0.0 and Fap0.1 in the output, because the interfaces are split across the ASICs). Only interfaces and VoQs with traffic are shown by the command:
lab#show platform fap voq scheduling VOQ scheduling mode: weighted-fair-queuing ingress rate per flow Ethernet22/1, TC7 Ingress Core Traffic Rate -------------------- -------------------- Fap0.0 40.000 bps Ethernet29/1, TC1 Ingress Core Traffic Rate -------------------- -------------------- Fap0.0 1.326 kbps Ethernet29/1, TC2 Ingress Core Traffic Rate -------------------- -------------------- Fap0.0 28.000 bps Ethernet29/1, TC3 Ingress Core Traffic Rate -------------------- -------------------- Fap0.0 12.000 bps Ethernet29/1, TC6 Ingress Core Traffic Rate -------------------- -------------------- Fap0.1 34.093 kbps Ethernet29/1, TC7 Ingress Core Traffic Rate -------------------- -------------------- Fap0.0 44.714 kbps Fap0.1 7.097 kbps
show platform sand voq tail-drop
This command shows the default buffer size allocated for each VoQ based on the port speed.
"MaxQueBuffSize" shows the number of memory descriptors which are allocated for a port of the given size. The memory descriptors being pointed to on Jericho are of size 1024 bytes in DRAM. The DRAM Buffer (4/8/12G) space is shared between all port VOQs. "MaxQueSize" is showing how many bytes these descriptors approximately total.
For example, the 50k buffers allocated for a 10G port equates to roughly 50MBs. However, each buffer can't hold more than a single packet. A stream of 10x 512 byte packets are stored in 10x buffers/descriptors. If a packet is larger than 1024 bytes it is split across multiple buffers. A stream of 10x 1500 byte packets would be stored in 20 buffers. This means that 50k buffers might be used up even though less than 50MBs of memory are being used (if all packets are < 1024 bytes). Also note that the buffers assigned to other port speeds are different sizes; the 5k buffers assigned for the 100Mbps port are 256 bytes each, the 500k buffers assigned to a 100G port are roughly 1468 bytes each.
These values can be changed per port speed per TC using "platform sand voq tail-drop speed100Gbps tc <0-7> ..."
lab#show platform sand voq tail-drop
Tail-Drop thresholds configuration:
----------------------------------
Speed DropPrec traffic- MaxQueSize MaxQueBuffSize
class (bytes) (buffers)
100Mbps Normal - 1310720 (1.25 MB) 5000
100Mbps Cpu - 1415577 (1.35 MB) 6000
1Gbps Normal - 13107200 (12.50 MB) 12500
1Gbps Cpu - 14155776 (13.50 MB) 13500
10Gbps Normal - 52428800 (50.00 MB) 50000
10Gbps Cpu - 53477376 (51.00 MB) 51000
25Gbps Normal - 131072000 (125.00 MB) 125000
25Gbps Cpu - 133693440 (127.50 MB) 127500
40Gbps Normal - 419430400 (400.00 MB) 200000
40Gbps Cpu - 423624704 (404.00 MB) 204000
50Gbps Normal - 419430400 (400.00 MB) 250000
50Gbps Cpu - 424673280 (405.00 MB) 255000
200Gbps Normal - 1048576000 (1000.00 MB) 700000
200Gbps Cpu - 1069547520 (1020.00 MB) 720000
400Gbps Normal - 2097152000 (2000.00 MB) 700000
400Gbps Cpu - 2139095040 (2040.00 MB) 740000
100Gbps Normal - 734003200 (700.00 MB) 500000
100Gbps Cpu - 744488960 (710.00 MB) 510000
show platform fap buffering egress
The max column shows the maximum number of packet descriptors that can be allocated by a VoQ for a specific port speed, and the FC column shows the threshold, when if the queue utilisation crosses, the ASIC starts to issue fewer tokens for packet transmission from ingress interface to egress interface, which leads to packet build-up at ingress interface (in ingress VoQ). As a result of packet build-up at ingress, if the congestion doesn't resolve, there will be tail-drop in that VoQ on the ingress interface:
lab#show platform fap buffering egress
Egress buffering settings for: Fap0
Egress buffering settings currently configured in hardware:
Max ports without head-of-line blocking:
100G : 2
10G : 8
200G : 1
25G : 8
400G : 1
40G : 4
50G : 4
Max full multicast queues per port : 3
name TC/DP Packet Descriptors 256B Data Buffers
Reserved max FC Reserved (MB) max (MB) FC (MB)
----------------------------------------------------------------------------------------------------------------
all - 65536 - - 24576(6.00) -
multicast - 53248 - - 12288(3.00) -
unicast - 12288 - - 12288(3.00) -
mcast SP0 all 12288 53248 - 1200 (0.29) 12288(3.00) -
TC0 - 40468 - 150 (0.04) 9338 (2.28) -
TC1 - 42065 - 150 (0.04) 9707 (2.37) -
TC2 - 43663 - 150 (0.04) 10076(2.46) -
TC3 - 45260 - 150 (0.04) 10444(2.55) -
TC4 - 46858 - 150 (0.04) 10813(2.64) -
TC5 - 48455 - 150 (0.04) 11182(2.73) -
TC6 - 50585 - 150 (0.04) 11673(2.85) -
TC7 - 53248 - 150 (0.04) 12288(3.00) -
mcast SP1 all - - - - - -
TC0 - - - - - -
TC1 - - - - - -
TC2 - - - - - -
TC3 - - - - - -
TC4 - - - - - -
TC5 - - - - - -
TC6 - - - - - -
TC7 - - - - - -
10G port (MC) - 6637 - - 1517 (0.37) -
10G queue (MC) all 160 - - - - -
DP0 - 1536 - - 342 (0.08) -
DP1 - 2085 - - 464 (0.11) -
DP2 - 1317 - - 293 (0.07) -
DP3 - 439 - - 97 (0.02) -
10G port (UC) - 12288 1000 - 12288(3.00) 1000 (0.24)
10G queue (UC) - 12288 333 - 12288(3.00) 333 (0.08)
25G port (MC) - 6637 - - 1517 (0.37) -
25G queue (MC) all 320 - - - - -
DP0 - 1536 - - 342 (0.08) -
DP1 - 2085 - - 464 (0.11) -
DP2 - 1317 - - 293 (0.07) -
DP3 - 439 - - 97 (0.02) -
25G port (UC) - 12288 1500 - 12288(3.00) 1500 (0.37)
25G queue (UC) - 12288 500 - 12288(3.00) 500 (0.12)
40G port (MC) - 13274 - - 3034 (0.74) -
40G queue (MC) all 800 - - - - -
DP0 - 3085 - - 695 (0.17) -
DP1 - 4187 - - 944 (0.23) -
DP2 - 2644 - - 596 (0.15) -
DP3 - 881 - - 198 (0.05) -
40G port (UC) - 12288 2250 - 12288(3.00) 2250 (0.55)
40G queue (UC) - 12288 750 - 12288(3.00) 750 (0.18)
50G port (MC) - 13274 - - 3034 (0.74) -
50G queue (MC) all 800 - - - - -
DP0 - 3085 - - 695 (0.17) -
DP1 - 4187 - - 944 (0.23) -
DP2 - 2644 - - 596 (0.15) -
DP3 - 881 - - 198 (0.05) -
50G port (UC) - 12288 2250 - 12288(3.00) 2250 (0.55)
50G queue (UC) - 12288 750 - 12288(3.00) 750 (0.18)
100G port (MC) - 26549 - - 6069 (1.48) -
100G queue (MC) all 1600 - - - - -
DP0 - 5733 - - 1404 (0.34) -
DP1 - 7781 - - 1905 (0.47) -
DP2 - 4914 - - 1203 (0.29) -
DP3 - 1638 - - 401 (0.10) -
100G port (UC) - 12288 3000 - 12288(3.00) 3000 (0.73)
100G queue (UC) - 12288 1000 - 12288(3.00) 1000 (0.24)
200G port (MC) - 53098 - - 12138(2.96) -
200G queue (MC) all 3200 - - - - -
DP0 - 5733 - - 2820 (0.69) -
DP1 - 7781 - - 3827 (0.93) -
DP2 - 4914 - - 2417 (0.59) -
DP3 - 1638 - - 805 (0.20) -
200G port (UC) - 12288 6000 - 12288(3.00) 6000 (1.46)
200G queue (UC) - 12288 2000 - 12288(3.00) 2000 (0.49)
400G port (MC) - 53098 - - 12138(2.96) -
400G queue (MC) all 3200 - - - - -
DP0 - 5733 - - 2820 (0.69) -
DP1 - 7781 - - 3827 (0.93) -
DP2 - 4914 - - 2417 (0.59) -
DP3 - 1638 - - 805 (0.20) -
400G port (UC) - 12288 8000 - 12288(3.00) 8000 (1.95)
400G queue (UC) - 12288 3000 - 12288(3.00) 3000 (0.73)
----------------------------------------------------------------------------------------------------------------
show platform fap interfaces counters
show platform fap voq delete
show cpu counters queue
show platform fap interrupts
show platform fap counters drops | nz
show hardware counter drop
show platform fap mapping
Previous page: 7280R3 - Jericho2 ASIC Drops
Next page: Control Plane MTU