7210 SAS-X QoS Overview

Scheduling

7210 SAS-X implements egress SAP scheduling and network port egress scheduling.
The list below specifies the various modes of scheduling supported:
• Egress SAP scheduling:
− FC based scheduling (without SAP aggregate shaper)
− SAP based scheduling (with SAP aggregate shaper)
• Network port egress scheduling - Only FC based scheduling is supported.
• Hybrid port egress Scheduling - Only FC based scheduling is supported.

 

FC Based Scheduling (without SAP aggregate shaper)

Two levels of scheduling occur at the SAP egress:
• Queue level
• Port egress rate or line rate level

At the queue level, the cir-level and pir-level determine the scheduling priority. The queue’s pir-weight determines the proportion of bandwidth given to the queue when multiple queues of the SAP at the same level are vying for available SAP bandwidth. The port egress rate determines the outgoing traffic rate at a given port.

SAP Based Scheduling (with SAP aggregate shaper for 7210 SAS-X devices)

Three levels of scheduling occur at the SAP egress:
• Queue level
• SAP aggregate shaper level
• Port egress rate or line rate level

At the queue level, the cir-level and pir-level determine the scheduling priority. The queue’s pir-weight determines the proportion of bandwidth given to the queue when multiple queues of the SAP at the same level are vying for available SAP bandwidth. The SAP aggregate shaper determines the bandwidth given to all FCs or queues belonging to the SAP. The CIR rate is set by system to the sum of the CIR rate of all queues that belong to the SAP. The port egress rate determines the outgoing traffic rate at a given port.

Scheduling Order when Multiple SAPs are Configured on a Port:

Pass	Scheduling Order
belowCIRpass	All SAPs configured on the same port are scheduled in round-robin fashion, sharing the available bandwidth equally.
aboveCIRpass	All SAPs configured on the same port are scheduled in round-robin fashion, sharing the available bandwidth equally.

Scheduling Order for Queues within an SAP:

Pass	Scheduling Order
belowCIRpass	The scheduler picks the queue with the highest cir-level among all the queues in use in a SAP and schedules it (that is, the cir-level “8” queues of the SAP are scheduled before cir-level “7” queues and so on.) until CIR of all the queues are met or available bandwidth is exhausted.
aboveCIRpass	The scheduler picks the queue with the highest cir-level among all the queues in use in a SAP and schedules it (that is, pir-level “4” queues of the SAP are scheduled before pir-level “3” queues and so on.) until PIR of all the queues are met or available bandwidth is exhausted.

 

Port Schedulers

The system creates a port scheduler for each of the ports (both access ports and network ports) in the system and distributes the available bandwidth to all the queues using that port to send out packets. The port scheduler works either at line-rate or configured egress rate. The port scheduler allocates bandwidth to the SAPs configured on the access port or to the queues configured on the network port.

The system creates a SAP scheduler for all the SAPs in the system. It distributes the available port bandwidth to all the queues allocated to a SAP. The SAP scheduler allocates bandwidth to the queues based on the configured CIR, PIR rates and the priority and weight assigned to the queues.

Listed below are the two passes made by the port scheduler:
• belowCIR pass
• aboveCIR pass

In the belowCIR pass, the port scheduler distributes the available port bandwidth among all the queues created on the port based on the configured cir-level, until the configured CIR rate is met. Queues with higher cir-level are scheduled first. The system services queues with CIR level 8 prior to servicing other queues in the system.

Note: To ensure that all the queues in the system are serviced by the scheduler, the queues with cir-level “8” must be capped to a rate, typically less than the total available port rate.

In the aboveCIR pass, the port scheduler distributes the remaining port bandwidth among all the queues based on its pir-level and pir-weight. The pir-level determines the priority of the queue in the aboveCIR pass. The pir-weight determines the proportion of bandwidth available for the queue when competing for bandwidth with other queues at the same level.

The pir-level assigned by the system is given below:

CIR Level Configured for use in CIR loop:	PIR level assigned by system for use in PIR loop:
8 (highest)	N/A
7	7
6	6
5	5
4	4
3	3
2	2
1 (lowest)	1 (lowest)

 

A queue configured with cir-level 8 in the system always get the configured CIR rate as long as bandwidth is available, irrespective of whether the CIR or PIR of other queues in the system are met. For a queue with the cir-level 8, the configured PIR is ignored by the system.

 

Committed Information Rate

The committed information rate (CIR) for a queue performs two distinct functions:
• Minimum bandwidth guarantees — Egress Queue’s CIR setting provides the bandwidth which will be given to this queue as compared to other queues on the port competing for a share of the available link bandwidth. The queue CIR guarantees bandwidth in all scenarios assuming sufficient link port bandwidth capacity is available. On 7210 platform the CIR rate cannot be oversubscribed. For each packet in an egress queue, the CIR is checked with the current transmission rate of the queue. If the current rate is at or below the CIR threshold, the queue is considered in-profile. If the current rate is above the threshold, the queue is considered out-of-profile. This in and out profile state of queue is linked to scheduler prioritizing behavior as discussed below.

• Scheduler queue priority metric — The scheduler serving a group of egress queues prioritizes individual queues based on their current CIR and PIR states. Queues operating below their CIR are always served before those queues operating at or above their CIR. Also see information about schedulers to know the scheduler behavior on different 7210 platforms.
Queues at the egress never marks the packets as in-profile or out-profile based on the queue CIR, PIR values. The in-profile and out-profile state of the queue interacts with the scheduler mechanism and provides the minimum and maximum bandwidth guarantees.

When defining the CIR for a queue, the value specified is the administrative CIR for the queue. The user has some control over how the administrative CIR is converted to an operational CIR should the hardware not support the exact CIR and PIR combination specified.

 

Peak Information Rate

The peak information rate (PIR) defines the maximum rate at which packets are allowed to exit the queue. It does not specify the maximum rate at which packets may enter the queue; this is governed by the queue's ability to absorb bursts. The actual transmission rate of a egress queue depends on more than just its PIR. Each queue is competing for transmission bandwidth with other queues. Each queue's PIR, CIR and the relative priority and/or weight of the scheduler serving the queue, all combine to affect a queue's ability to transmit packets.

The PIR is provisioned on egress queues within service egress QoS policies. The PIR for network queues are defined within network queue policies based on the forwarding class. The PIR for the queues is specified as a percentage of the network interface bandwidth.

When defining the PIR for a queue, the value specified is the administrative PIR for the queue.The user has some control over how the administrative PIR is converted to an operational PIR should the hardware not support the exact CIR and PIR values specified.

 

Queue Priority and Weight

On 7210 SAS-X, the priority for the queue can be specified by using the cir-level parameter. The system maps the cir-level to a pir-level and it is not user configurable. CIR-level parameter defines the scheduling priority of the queue in the CIR loop and the PIR loop (the system assigns the priority for the queue based on its cir-level).
CIR-level value of 8, represents the highest priority. Additionally, the scheduler always provides the configured bandwidth (CIR = PIR) for the queues assigned value (if bandwidth is available), irrespective of whether the CIR of other queues are met or not. In other words, CIR rate of level-8 queues in the system are provisioned before provisioning the CIR rate of queues at the other levels. PIR configured for queues at this level are ignored by the system. Users can also specify the weight for the queue. The weight parameter is used to determine the proportion of the available bandwidth that is allocated to the queues vying for bandwidth at the same priority.

On 7210 SAS-R6 and 7210 SAS-R12, the user has an option to specify whether the SAP egress queue is a strict queue or a weighted queue. The priority of the queue is determined by the FC of the queue, with higher priority FC being assigned a queue at a higher priority. The user can specify the weight of the queue for weighted queue. The weight parameter is used to determine the proportion of the available bandwidth that is allocated to the queues vying for bandwidth at the same priority.

 

Service Egress QoS Policies

Each queue in a policy is associated with one of the forwarding classes. Each queue can have an individual queue parameters allowing individual rate shaping of the forwarding class(es) mapped to the queue. The forwarding class determination per service egress packet is determined at ingress. If the packet ingressed the service on the same node, the service ingress classification rules determine the forwarding class of the packet. If the packet is received on a network port over a MPLS tunnel, the forwarding class is marked in the tunnel transport encapsulation (example - MPLS EXP bits).

On 7210 SAS-X, the FC to queue map is fixed, but the user has the flexibility to assign priority to a queue and change the scheduling order for that queue.

Default 7210-SAS-X FC-to-queue mappings for egress SAP or egress network port:

Traffic Class 		        FC Designation 	                Default Queue ID 	        DiffServ Name
Network Control 	        nc				queue 8 			NC2
High-1 				h1 				queue 7				NC1
Expedited			ef				queue 6 			EF
High-2				h2				queue 5 			AF4
Low-1				l1 				queue 4 			AF2
Assured				af				queue 3				AF1
Low-2				l2 				queue 2				CS1
Best-Effort			be 				queue 1				BE

 

 

Buffer Pools

The 7210 SAS-R6, 7210 SAS-R12 and 7210 SAS-X have single buffer pool per node, the system pool. All the queues created by the system are allocated buffers from this system pool. Queues come up with default buffers, and the buffers change accordingly when they are associated with a network port or SAP. Queue management policies allow the user to specify the parameters that determine buffer allocation to the queues. Buffer pools cannot be created or deleted in the 7210 SAS.

 

Queue Management Policies

Queue management policies allow the user to define the CBS, MBS and WRED parameters for use by the queue. The CBS and MBS parameters are used to allocate the appropriate amount of buffers from the system pool to the queues. The WRED parameters allow the user to define the WRED slope characteristics. User can define a high-slope and a low-slope for each of the queues. High-slope is used for in-profile packets being enqueued into the queues and low-priority slope is used for out-of- profile packets being enqueued into the queues.

By default each queue is associated with a default queue-management policy. The default policy allocates the appropriate amount of CBS and MBS buffers based on whether the queue is associated with a SAP or network port.

Note: If WRED is not configured, then tail drop is used.

By default, the high-priority and low-priority slopes are disabled.

 

Committed Burst Size and the Maximum Burst Size

The committed burst size and maximum burst size (CBS and MBS) parameters specify the amount of buffers reserved for a queue and up to how much of buffers a queue can contend for in the shared buffer space respectively. Once the reserved buffers for a given queue have been used, the queue contends with other queues for shared buffer resources up to the maximum burst size.

The CBS and MBS for the queues are configurable entities for the access and network ports and access uplink ports. The CBS and MBS value for the queues is set to appropriate default values which takes care of specific FC needs in terms of maintaining the differential treatment.

Default CBS Queues for 7210 SAS-X:
CBS (in KB) Network Queue: 750
CBS (in KB) Access Queue: 10

Default MBS Queues for 7210 SAS-X:
MBS (in KB) Network Queue: 1750
MBS (in KB) Access Queue: 65

 

default CBS == 10240 bytes (81,920 bits) == 0.00008192 seconds (>1ms) @ 1Gbps

 

Show the default queue management policy, that is has it's high and low profile WRED slopes disabled:

*A:switch1# show qos queue-mgmt default detail | match expression "Slop|Admin"
High Slope Parameters
Start Avg : 70 Admin State : Disabled
Low Slope Parameters
Start Avg : 50 Admin State : Disabled

Show that the default queue management policy is applied to a port with no other queue management policy configured:

*A:switch1# show qos queue-mgmt default detail | match 1/1/17 
- SAP : 1/1/17

Show that the CBS and MBS sizes for the port are the default values for an access port as per the documentation (10KBs and 65KBs respectively):

*A:switch1# show pools system 1/1/17

===============================================================================
Pool Information
===============================================================================
Port : 1/1/17 
Application : System 

MMU Total CBS : 45005 KB 


-------------------------------------------------------------------------------
ID FC-MAPS/ CBS (B) Depth A.CIR A.PIR
Queue MBS (B) O.CIR O.PIR
-------------------------------------------------------------------------------
Sap Ingress Queues:

No Match Found
Sap Egress Queues:

10105->1/1/17 be 10240 0 0 1000000
66560 0 1008000
10105->1/1/17 l2 10240 0 0 1000000
66560 0 1008000
10105->1/1/17 af 10240 0 0 1000000
66560 0 1008000
10105->1/1/17 l1 10240 0 0 1000000
66560 0 1008000
10105->1/1/17 h2 10240 0 0 1000000
66560 0 1008000
10105->1/1/17 ef 10240 0 0 1000000
66560 0 1008000
10105->1/1/17 h1 10240 0 0 1000000
66560 0 1008000
10105->1/1/17 nc 10240 0 0 1000000
66560 0 1008000
Network Queues:

No Match Found
===============================================================================

 

Buffers

The total device QoS buffer space if 1.3GBs. A maximum of 500MBs can be used for reserved buffer space, the remaining 800MBs is for shared buffer space.

- Maximum number of queues available in the device: 13320
- Max number of network queues (if all ports were network ports): 26 X 8 = 208
- Number of multipoint queues: 8
- Rest of the queues are available for access SAPs (13320 - 208 - 8 == 13104)

When the device comes up it reserves 10KB of CBS for each of 208 network queues and 13104 SAP queues.
12MBs is reserved for the system wide 8x multipoint queues.
With the default configuration all ports are network ports.
By default each network port requires approx 6 MB of reserved space (8 queues X 768 KB CBS per queue).
Thus the default reserved space for CBS is approx 300MB (26x6MB == 156MB for network ports + 12 MB for multipoint queues + 13104*10KB == 133MB for remaining access SAP queues).

The reserved buffer space for all queues is allowed to go up to a maximum of 500 MB with user configuration.
This means that by default the device has 200MBs free for CBS allocation.

The available buffer space for each queue over the CBS and up to its MBS limit is derived from the shared buffer space.
The default MBS setting is 66560 Bytes for each SAP queue and 1792 KB for each network queue.
The default MBS allocation for the 8 multipoint queues is ~42MBs.

The maximum configurable CBS or MBS for each queue is 500 MB.
Hence the maximum queue depth for an access SAP queue or a network queue is 500 MB.

 

The system does not allow to over subscribe CBS across all queues, MBS can be oversubscribed:

# All ports on swithc1 show 63415:
*A:switch1# show pools system 1/1/3 | match MMU 
MMU Total CBS : 63415 KB


# All ports on switch2 show 202255:
A:switch2# show pools system 1/1/8 | match MMU 
MMU Total CBS : 202255 KB


# All ports on switch3 show 116855:
*A:switch3# show pools system 1/1/8 | match MMU 
MMU Total CBS : 116855 KB

---