Date created: Friday, March 1, 2013 11:16:50 AM. Last modified: Wednesday, March 28, 2018 4:57:04 PM
CoPP & CPU Protection (IOS on 7600)
References:
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12-2SX/best/practices/recommendations.html#wp1214688
http://www.cisco.com/c/en/us/about/security-center/copp-best-practices.html#8
http://www.cisco.com/web/about/security/intelligence/coppwp_gs.html
http://www.cisco.com/c/en/us/td/docs/routers/7600/ios/12-2SR/configuration/guide/swcg/dos.html
http://www.cisco.com/c/en/us/support/docs/quality-of-service-qos/qos-congestion-management-queueing/18664-rtgupdates.html
http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-6500-series-switches/prod_white_paper0900aecd802ca5d6.pdf
http://www.cisco.com/c/en/us/support/docs/ip/access-lists/43920-iacl.html
http://www.cisco.com/c/dam/en/us/products/collateral/security/ios-network-foundation-protection-nfp/prod_white_paper0900aecd804ac831.pdf
http://www.cisco.com/c/en/us/td/docs/routers/7600/troubleshoot/guide/7600_Trouble_Shooting.pdf
https://tools.ietf.org/html/rfc6192
https://www.nanog.org/meetings/nanog42/presentations/Bhaiji_Network_Core.pdf
http://mailman.nanog.org/pipermail/nanog/2010-June/022649.html
https://puck.nether.net/pipermail/cisco-nsp/2015-June/099836.html
CoPP Overview:
To remove and reduce unwanted traffic from the CPU path "CoPP" alone is not enough. On platforms like the 7600/6500 they also have MLS hardware rate-limiters which must be configured to work alongside CoPP. The MLS HWRL are acting against all trafic transiting though the router as well as traffic destined for the router, were as CoPP is specifically on the punt path only so care must be taken with MLS HWRL not to negatively affect transit traffic.
Some traffic will hit the mls policers if for example no adjacency is found. When traffic hits the MLS rate limites it bypasses CoPP. Traffic passes through one or the other only, not both.Cards with DFCs perform CoPP in the line card. With CFCs the PFC provides the hardware CoPP processing.
Also features that cause traffic to originate from the CPU can be tuned like ICMP uncreachables.
Note Before: CoPP doesn't just cover traffic that is destined for the CPU, all traffic that is passed to the CPU will be affected by a CoPP policy, for example CPU punted traffic.
Note Before: Some devices can have issues when MPLS Explicit Null is used. The CoPP policies won't match traffic coming into the control plane if it has label 0 applied. 7600s are one platform that can handle MPLS explicit null though, it will look beyond the null label to match a traffic class.
Some bugs to be aware off on 7600s:
CSCsf25709 - "vpn-cam gets disable if class-default is in use in an applied policy-map (or a MATCH ANY clause is used within another class)".
CSCsi25255 - "policers using "class-default" reduce vpn forwarding performance".
Because the VPN-CAM is disable all L3 VPN traffic is passed through the policer!
CSCsf96383 - "After reload Control plane policing is not applied in hw anymore"
Check with "show mls qos ip | i CPP" that the QoS policies are applied to the control plane.
CSCsg85740 - "mls rate-limit mtu-failure drops >MTU-18 packets when CoPP configured"
Hardware Rate Limiter Example:
! There can be no CoPP in hardware or MLS rate-limtiters without first globally enabling MLS QoS mls qos
! ***********Interrupt/Process Scheduler*********** ! scheduler allocate ! default: scheduler allocate 4000 800 ! First integer, maximum number of microseconds to spend on fast switching within any one ! network interrupt context ! Second integer, guarantees the minimum number of microseconds to spend at the process ! level when network interrupts are disabled scheduler allocate 1000 2000 ! Decrease the maximum amount of continuous time the CPU may spend on any one process ! Default: process-max-time 200 (ms) process-max-time 100
! ***********ICMP Unreachables*********** ! Stop sending ICMP unreachable messages for traffic we black hole, default is to send ! ICMP unreachables interface null0 ! default: ip unreachables ! default: ipv6 unreachables no ipv6 unreachables no ip unreachables ! Rate-limit icmp unreachables code 0, network unreachable and code 4, fragmentation ! needed when DF bit set, default is 500ms for both, check before and after with ! "show ip icmp rate-limit" ip icmp rate-limit unreachable 1000 ip icmp rate-limit unreachable df 1000
! ***********Directred Broadcasts*********** ! Move directed broadcasts into hardware and not don't forward them to the RP ! Interface level command ! default: no mls ip directed-broadcast (config-if)# mls ip directed-broadcast exclude-router ! As of IOS 11, "no ip directed-broadcast" is the configured by default on all interfaces so the above mls command is not needed, only if directed broadcasts have been explicitly enabled on for an interface.
! ***********CoPP & MLS*********** ! CoPP is supported in software for multicast and broadcast traffic. ! Hardware support for multicast and broadcast traffic is provided by special-cases ! hardware-based rate-limiters. ! ARP traffic is not covered by CoPP (neither hardware nor software) at all on 6500/7600 ! platforms. However, an ARP special-cases hardware-based rate-limiter is available. ! The special-cases hardware-based rate-limiters will override the hardware CoPP policy for ! packets matching the rate-limiters criteria. ! CoPP uses hardware QoS TCAM resources, check TCAM usage before and after configuring CoPP with ! "show tcam counts" or "show tcam utilization"
! "show tcam interface fa5/2 acl in ip detail"
! "show mls statistics"
! CoPP does not support ACL entries with log keyword. ! Only the “input” direction is supported for CoPP.
! ***********MLS Limiters/Policers*********** ! Check MLS rate limiter usage before and after configuring with ! "show mls rate-limit" ! "show mls rate-limit usage" ! "clear icmp rate-limit"
! This command is basically useless and shouldn't be used:
! mls rate-limit unicast cef receive XXX
! "Rate limits all packets that contain any route processor IP address as the destination address"
! HWRLs take precedence over CoPP so this will rate-limit RP traffic without allowing a CoPP policy to properly filter it.
! mls qos protocol xxx mls qos protocol arp police 2000000 62000 mls qos protocol neigh-discover police 2000000 62000
! Layer 3 HWRLs:
! Disable this rate-limit which is used by default (unless you need it)
! because it uses one of the HWRL spaces in CEF
no mls rate-limit unicast acl vacl-log
! mls rate-limit multicast ipv4 xxx mls rate-limit multicast ipv4 fib-miss 2000 10 !Note that this rate-limiter uses a special register that is not accounted for in the available ten hardware registers and it is applied globally, not on a per-forwarding-engine basis. mls rate-limit multicast ipv4 non-rpf 10 10 mls rate-limit multicast ipv4 partial 2000 10
mls rate-limit multicast ipv4 ip-options 10 10
! mls rate-limit multicast ipv6 xxx
! Several HWRL for Multicast don't work on 6500s:
! https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuo37358/?referring_site=bugquickviewredir
! broken: mls rate-limit multicast ipv6 connected 1500 20
! broken: mls rate-limit multicast ipv6 default-drop 1000 20
! broken: mls rate-limit multicast ipv6 mld 10 1
! broken: mls rate-limit multicast ipv6 route-cntl 10 1
! mls rate-limit unicast ip xxx
mls rate-limit unicast ip features 10 10 mls rate-limit unicast ip icmp redirect 0 ! These next four all share a single HWRL mls rate-limit unicast ip icmp unreachable no-route 10 10 mls rate-limit unicast ip icmp unreachable acl-drop 10 10 mls rate-limit unicast ip rpf-failure 10 10 mls rate-limit unicast ip errors 10 10
! mls rate-limit unicast ip options 10 10
! This one is not enabled because it is covered in the CoPP polocy
! mls rate-limit unicast cef xxx mls rate-limit unicast cef glean 200 50 ! "all" applied to both Unicast and Multicast mls rate-limit all ttl-failure 200 50 mls rate-limit all mtu-failure 10 10 ! Was affected by CSCsg85740 but fixed now
! Layer 2 HWRLs:
! mls rate-limit layer2 xxx mls rate-limit layer2 pdu 20 20
mls rate-limit multicast ipv4 igmp 2000 10
! Before:
abr1#show mls rate-limit
Sharing Codes: S - static, D - dynamic
Codes dynamic sharing: H - owner (head) of the group, g - guest of the group
Rate Limiter Type Status Packets/s Burst Sharing
--------------------- ---------- --------- ----- -------
MCAST NON RPF Off - - -
MCAST DFLT ADJ On 100000 100 Not sharing
MCAST DIRECT CON Off - - -
ACL BRIDGED IN Off - - -
ACL BRIDGED OUT Off - - -
IP FEATURES Off - - -
ACL VACL LOG On 2000 1 Not sharing
CEF RECEIVE Off - - -
CEF GLEAN Off - - -
MCAST PARTIAL SC On 100000 100 Not sharing
IP RPF FAILURE On 100 10 Group:0 S
TTL FAILURE On 97 10 Not sharing
ICMP UNREAC. NO-ROUTE On 100 10 Group:0 S
ICMP UNREAC. ACL-DROP On 100 10 Group:0 S
ICMP REDIRECT Off - - -
MTU FAILURE On 997 10 Not sharing
MCAST IP OPTION Off - - -
UCAST IP OPTION Off - - -
LAYER_2 PDU Off - - -
LAYER_2 PT Off - - -
DHCP Snooping IN Off - - -
DHCP Snooping OUT Off - - -
ARP Inspection Off - - -
LAYER_2 PORTSEC Off - - -
LAYER_2 MiniProto Off - - -
IP ERRORS On 100 10 Group:0 S
CAPTURE PKT Off - - -
MCAST IGMP Off - - -
MCAST IPv6 DIRECT CON Off - - -
MCAST IPv6 ROUTE CNTL Off - - -
MCAST IPv6 *G M BRIDG Off - - -
MCAST IPv6 SG BRIDGE Off - - -
MCAST IPv6 DFLT DROP Off - - -
MCAST IPv6 SECOND. DR Off - - -
MCAST IPv6 *G BRIDGE Off - - -
MCAST IPv6 MLD Off - - -
IP ADMIS. ON L2 PORT Off - - -
IPV6 FIRST HOP SECURI Off - - -
UCAST IP TINY FRAG Off - - -
MCAST IP TINY FRAG Off - - -
LAYER_2 MACSEC Off - - -
MCAST IPv4 PIM Off - - -
IPV6 BRIDGE Off - - -
abr1#show mls rate-limit usage
Rate Limiter Type Packets/s Burst
--------------------- --------- -----
Layer3 Rate Limiters:
RL# 0: Free - - -
RL# 1: Free - - -
RL# 2: Free - - -
RL# 3: Used
MTU FAILURE 997 10
RL# 4: Used
TTL FAILURE 97 10
RL# 5: Used
IP RPF FAILURE 100 10
ICMP UNREAC. NO-ROUTE 100 10
ICMP UNREAC. ACL-DROP 100 10
IP ERRORS 100 10
RL# 6: Used
ACL VACL LOG 2000 1
RL# 7: Used
MCAST DFLT ADJ 100000 100
RL# 8: Rsvd for capture - - -
Layer2 Rate Limiters:
RL# 9: Reserved
RL#10: Reserved
RL#11: Free - - -
RL#12: Free - - -
! After
abr1#show mls rate-limit
Sharing Codes: S - static, D - dynamic
Codes dynamic sharing: H - owner (head) of the group, g - guest of the group
Rate Limiter Type Status Packets/s Burst Sharing
--------------------- ---------- --------- ----- -------
MCAST NON RPF On 10 10 Not sharing
MCAST DFLT ADJ On 2000 10 Not sharing
MCAST DIRECT CON Off - - -
ACL BRIDGED IN Off - - -
ACL BRIDGED OUT Off - - -
IP FEATURES On 10 10 Not sharing
ACL VACL LOG Off - - -
CEF RECEIVE Off - - -
CEF GLEAN On 200 50 Not sharing
MCAST PARTIAL SC On 2000 10 Not sharing
IP RPF FAILURE On 10 10 Group:0 S
TTL FAILURE On 200 50 Not sharing
ICMP UNREAC. NO-ROUTE On 10 10 Group:0 S
ICMP UNREAC. ACL-DROP On 10 10 Group:0 S
ICMP REDIRECT On 0 0 -
MTU FAILURE On 10 10 Not sharing
MCAST IP OPTION On 10 10 Group:3 S
UCAST IP OPTION Off - - -
LAYER_2 PDU On 20 20 Not sharing
LAYER_2 PT Off - - -
DHCP Snooping IN Off - - -
DHCP Snooping OUT Off - - -
ARP Inspection Off - - -
LAYER_2 PORTSEC Off - - -
LAYER_2 MiniProto Off - - -
IP ERRORS On 10 10 Group:0 S
CAPTURE PKT Off - - -
MCAST IGMP On 2000 10 Not sharing
MCAST IPv6 DIRECT CON Off - - -
MCAST IPv6 ROUTE CNTL Off - - -
MCAST IPv6 *G M BRIDG Off - - -
MCAST IPv6 SG BRIDGE Off - - -
MCAST IPv6 DFLT DROP Off - - -
MCAST IPv6 SECOND. DR Off - - -
MCAST IPv6 *G BRIDGE Off - - -
MCAST IPv6 MLD Off - - -
IP ADMIS. ON L2 PORT Off - - -
IPV6 FIRST HOP SECURI Off - - -
UCAST IP TINY FRAG Off - - -
MCAST IP TINY FRAG On/Sharing 10 10 -
LAYER_2 MACSEC Off - - -
MCAST IPv4 PIM Off - - -
IPV6 BRIDGE Off - - -
abr1#show mls rate-limit usage
Rate Limiter Type Packets/s Burst
--------------------- --------- -----
Layer3 Rate Limiters:
RL# 0: Used
MCAST IP OPTION 10 10
RL# 1: Used
CEF GLEAN 200 50
RL# 2: Used
MCAST NON RPF 10 10
RL# 3: Used
MTU FAILURE 10 10
RL# 4: Used
TTL FAILURE 200 50
RL# 5: Used
IP RPF FAILURE 10 10
ICMP UNREAC. NO-ROUTE 10 10
ICMP UNREAC. ACL-DROP 10 10
IP ERRORS 10 10
RL# 6: Used
IP FEATURES 10 10
RL# 7: Used
MCAST DFLT ADJ 2000 10
RL# 8: Rsvd for capture - - -
Layer2 Rate Limiters:
RL# 9: Reserved
RL#10: Reserved
RL#11: Used
LAYER_2 PDU 20 20
RL#12: Used
MCAST IGMP 2000 10
Example CoPP config below. Use "show tcp brief all | i LIST" and "show udp" to check that all TCP and UDP sessions to the RP would be sovered by the policy being written.
! ***********Access-groups*********** !!!!! Control plane traffic (such as routing protocols) ip access-list extended CoPP-Limit-and-Permit-BGP permit tcp any eq 179 any permit tcp any any eq 179 ipv6 access-list CoPP-Limit-and-Permit-BGPv6 permit tcp any eq 179 any permit tcp any any eq 179 ip access-list extended CoPP-Limit-and-Permit-RSVP permit 46 any any ip access-list extended CoPP-Limit-and-Permit-LDP permit tcp any any eq 646 permit tcp any eq 646 any permit udp any any eq 646 permit udp any eq 646 any ip access-list extended CoPP-Limit-and-Permit-OSPF permit ospf any any ipv6 access-list CoPP-Limit-and-Permit-OSPFv3 permit 89 any any ip access-list extended CoPP-Limit-and-Permit-HSRP permit udp host 224.0.0.2 eq 1985 any permit udp any host 224.0.0.2 eq 1985 permit udp host 224.0.0.102 eq 1985 any permit udp any host 224.0.0.102 eq 1985 ip access-list extended CoPP-Limit-and-Permit-BFD permit udp any any eq 3784 permit udp any eq 3784 any !!!!! Control plane SYN traffic ip access-list extended CoPP-Limit-and-Permit-BGP-SYN permit tcp any eq 179 any syn permit tcp any any eq 179 syn ipv6 access-list CoPP-Limit-and-Permit-BGPv6-SYN permit tcp any eq 179 any syn permit tcp any any eq 179 syn ip access-list extended CoPP-Limit-and-Permit-LDP-SYN permit tcp any any eq 646 syn permit tcp any eq 646 any syn !!!!! Management plane traffic (such as SSH,SNMP) ip access-list extended CoPP-Limit-and-Permit-TACACS permit tcp host 192.168.30.50 eq 49 any permit tcp any host 192.168.30.50 eq 49 permit tcp host 192.168.30.51 eq 49 any permit tcp any host 192.168.30.51 eq 49 permit udp host 192.168.30.50 eq 49 any permit udp any host 192.168.30.50 eq 49 permit udp host 192.168.30.51 eq 49 any permit udp any host 192.168.30.51 eq 49 ip access-list extended CoPP-Limit-and-Permit-RADIUS permit udp host 192.168.30.50 eq 1812 1813 any permit udp any host 192.168.30.50 eq 1812 1813 permit udp host 192.168.30.51 eq 1812 1813 any permit udp any host 192.168.30.51 eq 1812 1813 ip access-list extended CoPP-Limit-and-Permit-TELNET-SSH permit tcp 192.168.30.0 0.0.1.255 any eq 22 permit tcp any eq 22 192.168.30.0 0.0.1.255 permit tcp 192.168.30.0 0.0.1.255 any eq 21 permit tcp any eq 21 192.168.30.0 0.0.1.255 ip access-list extended CoPP-Limit-and-Permit-SNMP permit udp 192.168.30.0 0.0.1.255 any eq snmp snmptrap permit udp any eq 161 161 192.168.30.0 0.0.1.255 ip access-list standard CoPP-Limit-and-Permit-NTP permit udp any host 192.168.30.20 eq 123 permit udp host 192.168.30.20 eq 123 any permit udp any host 192.168.30.22 eq 213 permit udp host 192.168.30.22 eq 123 any ip access-list standard CoPP-Limit-and-Permit-SNMP-Pollers permit udp host 192.168.30.34 any eq 161 permit udp host 192.168.30.35 any eq 161 permit udp host 192.168.30.33 any eq 161 permit udp host 192.168.30.40 any eq 161 permit udp any eq 161 host 192.168.30.34 permit udp any eq 161 host 192.168.30.35 permit udp any eq 161 host 192.168.30.33 permit udp any eq 161 host 192.168.30.40 !!!!! Management SYN plane traffic ip access-list extended CoPP-Limit-and-Permit-TACACS-SYN permit tcp host 192.168.30.50 eq 49 any syn permit tcp any host 192.168.30.50 eq 49 syn permit tcp host 192.168.30.51 eq 49 any syn permit tcp any host 192.168.30.51 eq 49 syn ip access-list extended CoPP-Limit-and-Permit-TELNET-SSH-SYN permit tcp 192.168.30.0 0.0.1.255 any eq 22 permit tcp any eq 22 192.168.30.0 0.0.1.255 permit tcp 192.168.30.0 0.0.1.255 any eq 23 permit tcp any eq 23 192.168.30.0 0.0.1.255 !!!!! Forwarding traffic that is CPU punted ip access-list extended CoPP-Limit-and-Permit-IP-Options permit ip any any option record-route ip access-list extended CoPP-Limit-and-Permit-ICMP permit icmp any any echo permit icmp any any echo-request permit icmp any any unreachable permit icmp any any ttl-exceeded permit icmp any any packet-too-big ipv6 access-list CoPP-Limit-and-Permit-ICMPv6 permit icmp any any echo-reply permit icmp any any echo-request permit icmp any any unreachable permit icmp any any ttl-exceeded permit icmp any any packet-too-big ip access-list extended CoPP-Limit-and-Permit-UDP-Traceroute permit udp any any range 33434 33689 permit udp any range 33434 33689 any ip access-list extended CoPP-Limit-and-Permit-TCP-syn-fin-rst permit tcp any any fin syn rst ipv6 access-list CoPP-Limit-and-Permit-TCP-syn-fin-rst-IPv6 permit tcp any any fin syn rst ip access-list extended CoPP-Limit-and-Permit-TCP-established permit tcp any any established ipv6 access-list CoPP-Limit-and-Permit-TCP-established-IPv6 permit tcp any any established !!!!! Traffic we want to always drop ip access-list extended CoPP-Deny-IP-Fragments permit ip any any fragments permit icmp any any fragments ip access-list CoPP-Deny-IPv6-Fragments permit ip any any fragments permit icmp any any fragments ip access-list extended CoPP-Deny-ICMP permit icmp any any ipv6 access-list CoPP-Deny-ICMPv6 permit icmp any any ip access-list extended CoPP-Deny-IP-Options permit ip any any option any ip access-list extended CoPP-Deny-IGMP permit igmp any any !!!!! Catch-all for traffic that doesn't match any of the above ACLS ip access-list extended CoPP-Limit-and-Permit-Catch-All permit icmp any any permit ip any any ipv6 access-list CoPP-Limit-and-Permit-Catch-All-IPv6 permit icmp any any permit ipv6 any any
! ***********Class Definitions*********** !!!!! Control plane traffic (such as routing protocols) class-map match-any CoPP-Limit-and-Permit-Critical match access-group name CoPP-Limit-and-Permit-BGP match access-group name CoPP-Limit-and-Permit-BGPv6 match access-group name CoPP-Limit-and-Permit-RSVP match access-group name CoPP-Limit-and-Permit-LDP match access-group name CoPP-Limit-and-Permit-LDP match access-group name CoPP-Limit-and-Permit-OSPF match access-group name CoPP-Limit-and-Permit-OSPFv3 match access-group name CoPP-Limit-and-Permit-HSRP match access-group name CoPP-Limit-and-Permit-BFD !!!!! Control plane SYN traffic class-map match-any CoPP-Limit-and-Permit-Critical-SYN match access-group name CoPP-Limit-and-Permit-BGP-SYN match access-group name CoPP-Limit-and-Permit-BGPv6-SYN match access-group name CoPP-Limit-and-Permit-LDP-SYN !!!!! Management plane traffic (such as SSH,SNMP) class-map match-any CoPP-Limit-and-Permit-Management-Plane match access-group name CoPP-Limit-and-Permit-TACACS match access-group name CoPP-Limit-and-Permit-RADIUS match access-group name CoPP-Limit-and-Permit-TELNET-SSH match access-group name CoPP-Limit-and-Permit-SNMP match access-group name CoPP-Limit-and-Permit-NTP match access-group name CoPP-Limit-and-Permit-SNMP-Pollers !!!!! Management plane SYN traffic class-map match-any CoPP-Limit-and-Permit-Management-Plane-SYN match access-group name CoPP-Limit-and-Permit-TACACS-SYN match access-group name CoPP-Limit-and-Permit-TELNET-SSH-SYN !!!!! Forwarding traffic that is CPU punted class-map match-any CoPP-Limit-and-Permit-Forwarding-Plane match access-group name CoPP-Limit-and-Permit-IP-Options match access-group name CoPP-Limit-and-Permit-ICMP match access-group name CoPP-Limit-and-Permit-ICMPv6 match access-group name CoPP-Limit-and-Permit-UDP-Traceroute match access-group name CoPP-Limit-and-Permit-TCP-syn-fin-rst match access-group name CoPP-Limit-and-Permit-TCP-syn-fin-rst-IPv6 match access-group name CoPP-Limit-and-Permit-TCP-established match access-group name CoPP-Limit-and-Permit-TCP-established-IPv6 !!!!! Traffic we want to always drop class-map match-any CoPP-Deny-Always match access-group name CoPP-Deny-IGMP match access-group name CoPP-Deny-IP-Fragments match access-group name CoPP-Deny-IPv6-Fragments match access-group name CoPP-Deny-ICMP match access-group name CoPP-Deny-ICMPv6 !!!!! Catch-all for traffic that doesn't match any of the above ACLS class-map match-any CoPP-Catch-All match access-group name CoPP-Limit-and-Permit-Catch-All match access-group name CoPP-Limit-and-Permit-Catch-All-IPv6
! ***********Class Limiting*********** policy-map Control-Plane-Filter-In !!!!! Control plane traffic (such as routing protocols) class CoPP-Limit-and-Permit-Critical police cir 10000000 bc 312500 be 312500 conform-action transmit exceed-action drop violate-action drop !!!!! Control plane SYN traffic class CoPP-Limit-and-Permit-Critical-SYN police cir 250000 bc 7812 be 7812 conform-action transmit exceed-action drop violate-action drop !!!!! Management plane traffic (such as SSH,SNMP) class CoPP-Limit-and-Permit-Management-Plane police cir 1000000 bc 312500 be 312500 conform-action transmit exceed-action drop violate-action drop !!!!! Management plane SYN traffic class CoPP-Limit-and-Permit-Management-Plane-SYN police cir 250000 bc 3125 be 3125 conform-action transmit exceed-action drop violate-action drop !!!!! Forwarding traffic that is CPU punted class CoPP-Limit-and-Permit-Forwarding-Plane police cir 1000000 bc 31250 be 31250 conform-action transmit exceed-action drop violate-action drop !!!!! Traffic we want to always drop class CoPP-Deny-Always drop !!!!! Catch-all for traffic that doesn't match any of the above ACLS class CoPP-Catch-All police cir 500000 bc 15625 be 15625 conform-action transmit exceed-action drop violate-action drop !!!!! Any other non-IP traffic such as IS-IS (not being used here) or L2 keepalives could be cought with a class-default ! however for many paltforms this breaks a lot of stuff so it's not recommended.... ! class class-default ! police cir 1000000 bc 31250 be 31250 conform-action transmit exceed-action drop violate-action drop control-plane service-policy input Control-Plane-Filter-In
These are the outputs from applying CoPP
abr1#show mls qos ip
QoS Summary [IPv4]: (* - shared aggregates, Mod - switch module)
Int Mod Dir Class-map DSCP Agg Trust Fl AgForward-By AgPoliced-By
Id Id
-------------------------------------------------------------------------------
CPP 6 In CoPP-Limit 0 14 dscp 0 594716644 0
CPP 6 In CoPP-Limit 0 15 dscp 0 102965249 0
CPP 6 In CoPP-Limit 0 16 dscp 0 158932401 0
CPP 6 In CoPP-Deny- 0 17 dscp 0 28080 0
CPP 6 In CoPP-Catch 0 18 dscp 0 0 0
CPP 6 In CoPP-Limit 0 19 dscp 0 0 0
CPP 6 In CoPP-Limit 0 20 dscp 0 0 0
Vl3002 6 In class-defa 46 7 No 0 0 0
Gi1/2 6 Out PSN-Real-T 0 8 -- 0 0 0
Gi1/2 6 Out PSN-Applic 0 9 -- 0 0 0
Gi1/2 6 Out PSN-Applic 0 10 -- 0 125693 15818
Gi1/2 6 Out PSN-Applic 0 11 -- 0 0 0
Gi1/2 6 Out PSN-Applic 0 12 -- 0 6088 0
Gi1/2 6 Out class-defa 0 13 -- 0 27365727626 1969338894
Vl104 6 Out PSN-Real-T 0 1 -- 0 120806022 0
Vl104 6 Out PSN-Applic 0 2 -- 0 0 0
Vl104 6 Out PSN-Applic 0 3 -- 0 930976 0
Vl104 6 Out PSN-Applic 0 4 -- 0 0 0
Vl104 6 Out PSN-Applic 0 5 -- 0 1856 0
Vl104 6 Out class-defa 0 6 -- 0 2259866025 4922990
All 6 - Default 0 0* No 0 262775558861 0
abr1#show vlan internal usage | i Control
4087 Control Plane Protection
abr1#remote command switch show tcam interface vlan 4087 qos type2 ip
* Global Defaults shared
------------------------------------------------------
QOS Results:
A - Aggregate Policing F - Microflow Policing
M - Mark T - Trust
U - Untrust
------------------------------------------------------
MAU any any
MAU ospf any any
MAU tcp any eq bgp any
MAU tcp any eq 646 any
MAU udp any eq 646 any
MAU udp any eq 3784 any
MAU tcp any any eq bgp
MAU tcp any any eq 646
MAU udp any any eq 646
MAU udp any any eq 3784
MAU udp host 224.0.0.2 eq 1985 any
MAU udp host 224.0.0.102 eq 1985 any
MAU udp any host 224.0.0.2 eq 1985
MAU udp any host 224.0.0.102 eq 1985
MAU tcp 192.168.30.0 0.0.1.255 any eq 22
MAU tcp 192.168.30.0 0.0.1.255 any eq ftp
MAU udp 192.168.30.0 0.0.1.255 any eq snmp
MAU udp 192.168.30.0 0.0.1.255 any eq snmptrap
MAU tcp host 192.168.30.50 eq tacacs any
MAU tcp host 192.168.30.51 eq tacacs any
MAU udp host 192.168.30.50 eq tacacs any
MAU udp host 192.168.30.51 eq tacacs any
MAU udp host 192.168.30.50 eq 1812 any
MAU udp host 192.168.30.50 eq 1813 any
MAU udp host 192.168.30.51 eq 1812 any
MAU udp host 192.168.30.51 eq 1813 any
MAU udp host 192.168.30.20 eq ntp any
MAU udp host 192.168.30.22 eq ntp any
MAU tcp any eq 22 192.168.30.0 0.0.1.255
MAU tcp any eq ftp 192.168.30.0 0.0.1.255
MAU udp any eq snmp 192.168.30.0 0.0.1.255
MAU tcp any host 192.168.30.50 eq tacacs
MAU tcp any host 192.168.30.51 eq tacacs
MAU udp any host 192.168.30.50 eq tacacs
MAU udp any host 192.168.30.51 eq tacacs
MAU udp any host 192.168.30.50 eq 1812
MAU udp any host 192.168.30.50 eq 1813
MAU udp any host 192.168.30.51 eq 1812
MAU udp any host 192.168.30.51 eq 1813
MAU udp any host 192.168.30.20 eq ntp
MAU udp any host 192.168.30.22 eq 213
MAU ip any any
AT ip any any
At first the idea that the Software Counters are increasing might seem like a mistake in the ACLs that has caused the CPU destined traffic to also be rate-limited in software (or similar config mistake) however that is not the case. Expected traffic coming into the CPU (such as BGP updates for example) causes the software counters to increase because that traffic is supposed to be processed in software. What we are looking for here is traffic classes where the hardware counters are increasing much faster than the software counters to indicate they are either set too low, or policing a burst of traffic during an attack or network issue for example.
As long as the violations are not increasing at a high rate no traffic that should be sent to the CPU is being restricted. The hardware rate limiters are just that, programmed into the line cards so if during an attack the hardware policers are being hammered then they are doing their job and that’s fine, as long as for example BGP traffic isn’t being completely starved.
abr1#show policy-map control-plane
Control Plane
Service-policy input: Control-Plane-Filter-In
Hardware Counters:
class-map: CoPP-Limit-and-Permit-Critical (match-any)
Match: access-group name CoPP-Limit-and-Permit-BGP
Match: access-group name CoPP-Limit-and-Permit-BGPv6
Match: access-group name CoPP-Limit-and-Permit-RSVP
Match: access-group name CoPP-Limit-and-Permit-LDP
Match: access-group name CoPP-Limit-and-Permit-OSPF
Match: access-group name CoPP-Limit-and-Permit-OSPFv3
Match: access-group name CoPP-Limit-and-Permit-HSRP
Match: access-group name CoPP-Limit-and-Permit-BFD
police :
10000000 bps 312000 limit 312000 extended limit
Earl in slot 6 :
51302250422 bytes
5 minute offered rate 85976 bps
aggregate-forwarded 51302250422 bytes action: transmit
exceeded 0 bytes action: transmit
aggregate-forward 74832 bps exceed 0 bps
Software Counters:
Class-map: CoPP-Limit-and-Permit-Critical (match-any)
376908484 packets, 32494165377 bytes
5 minute offered rate 55000 bps, drop rate 0000 bps
Match: access-group name CoPP-Limit-and-Permit-BGP
324057526 packets, 28695202303 bytes
5 minute rate 49000 bps
Match: access-group name CoPP-Limit-and-Permit-BGPv6
181642 packets, 15090592 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-RSVP
8072 packets, 1276824 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-LDP
15000928 packets, 1138418002 bytes
5 minute rate 1000 bps
Match: access-group name CoPP-Limit-and-Permit-OSPF
6731250 packets, 731170868 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-OSPFv3
0 packets, 0 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-HSRP
30929060 packets, 1913006504 bytes
5 minute rate 2000 bps
Match: access-group name CoPP-Limit-and-Permit-BFD
4 packets, 284 bytes
5 minute rate 0 bps
police:
cir 10000000 bps, bc 312500 bytes, be 312500 bytes
conformed 376908486 packets, 32494165377 bytes; actions:
transmit
exceeded 0 packets, 0 bytes; actions:
transmit
violated 0 packets, 0 bytes; actions:
drop
conformed 55000 bps, exceeded 0000 bps, violated 0000 bps
Hardware Counters:
class-map: CoPP-Limit-and-Permit-Management-Plane (match-any)
Match: access-group name CoPP-Limit-and-Permit-TACACS
Match: access-group name CoPP-Limit-and-Permit-RADIUS
Match: access-group name CoPP-Limit-and-Permit-TELNET-SSH
Match: access-group name CoPP-Limit-and-Permit-SNMP
Match: access-group name CoPP-Limit-and-Permit-NTP
Match: access-group name CoPP-Limit-and-Permit-SNMP-Pollers
police :
1000000 bps 312000 limit 312000 extended limit
Earl in slot 6 :
9305000717 bytes
5 minute offered rate 9896 bps
aggregate-forwarded 9305000717 bytes action: transmit
exceeded 0 bytes action: transmit
aggregate-forward 2240 bps exceed 0 bps
Software Counters:
Class-map: CoPP-Limit-and-Permit-Management-Plane (match-any)
77496404 packets, 8995276492 bytes
5 minute offered rate 5000 bps, drop rate 0000 bps
Match: access-group name CoPP-Limit-and-Permit-TACACS
30642 packets, 1863750 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-RADIUS
0 packets, 0 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-TELNET-SSH
480489 packets, 36072767 bytes
5 minute rate 2000 bps
Match: access-group name CoPP-Limit-and-Permit-SNMP
76917195 packets, 8951213045 bytes
5 minute rate 4000 bps
Match: access-group name CoPP-Limit-and-Permit-NTP
68077 packets, 6126930 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-SNMP-Pollers
0 packets, 0 bytes
5 minute rate 0 bps
police:
cir 1000000 bps, bc 312500 bytes, be 312500 bytes
conformed 77496404 packets, 8995276492 bytes; actions:
transmit
exceeded 0 packets, 0 bytes; actions:
transmit
violated 0 packets, 0 bytes; actions:
drop
conformed 5000 bps, exceeded 0000 bps, violated 0000 bps
Hardware Counters:
class-map: CoPP-Limit-and-Permit-Forwarding-Plane (match-any)
Match: access-group name CoPP-Limit-and-Permit-IP-Options
Match: access-group name CoPP-Limit-and-Permit-ICMP
Match: access-group name CoPP-Limit-and-Permit-ICMPv6
Match: access-group name CoPP-Limit-and-Permit-UDP-Traceroute
Match: access-group name CoPP-Limit-and-Permit-TCP-syn-fin-rst
Match: access-group name CoPP-Limit-and-Permit-TCP-syn-fin-rst-IPv6
Match: access-group name CoPP-Limit-and-Permit-TCP-established
Match: access-group name CoPP-Limit-and-Permit-TCP-established-IPv6
police :
1000000 bps 31000 limit 31000 extended limit
Earl in slot 6 :
16948565405 bytes
5 minute offered rate 23264 bps
aggregate-forwarded 16948565405 bytes action: transmit
exceeded 0 bytes action: transmit
aggregate-forward 23320 bps exceed 0 bps
Software Counters:
Class-map: CoPP-Limit-and-Permit-Forwarding-Plane (match-any)
85359674 packets, 6650725463 bytes
5 minute offered rate 7000 bps, drop rate 0000 bps
Match: access-group name CoPP-Limit-and-Permit-IP-Options
36055 packets, 3245310 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-ICMP
79493442 packets, 6241343837 bytes
5 minute rate 7000 bps
Match: access-group name CoPP-Limit-and-Permit-ICMPv6
0 packets, 0 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-UDP-Traceroute
4408893 packets, 311453581 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-TCP-syn-fin-rst
1390938 packets, 89258079 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-TCP-syn-fin-rst-IPv6
0 packets, 0 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-TCP-established
30346 packets, 5424734 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-TCP-established-IPv6
0 packets, 0 bytes
5 minute rate 0 bps
police:
cir 1000000 bps, bc 31250 bytes, be 31250 bytes
conformed 85346535 packets, 6631803600 bytes; actions:
transmit
exceeded 12489 packets, 17964959 bytes; actions:
transmit
violated 651 packets, 956982 bytes; actions:
drop
conformed 7000 bps, exceeded 0000 bps, violated 0000 bps
Hardware Counters:
class-map: CoPP-Deny-Always (match-any)
Match: access-group name CoPP-Deny-IGMP
Match: access-group name CoPP-Deny-IP-Fragments
Match: access-group name CoPP-Deny-IPv6-Fragments
Match: access-group name CoPP-Deny-ICMP
Match: access-group name CoPP-Deny-ICMPv6
police :
8000 bps 1000 limit 1000 extended limit
Earl in slot 6 :
2585914 bytes
5 minute offered rate 0 bps
aggregate-forwarded 2585914 bytes action: transmit
exceeded 0 bytes action: transmit
aggregate-forward 0 bps exceed 0 bps
Software Counters:
Class-map: CoPP-Deny-Always (match-any)
1285120 packets, 115082280 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: access-group name CoPP-Deny-IGMP
0 packets, 0 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Deny-IP-Fragments
0 packets, 0 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Deny-IPv6-Fragments
0 packets, 0 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Deny-ICMP
103 packets, 9706 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Deny-ICMPv6
1285017 packets, 115072574 bytes
5 minute rate 0 bps
police:
cir 8000 bps, bc 1500 bytes, be 1500 bytes
conformed 1285120 packets, 115082280 bytes; actions:
transmit
exceeded 0 packets, 0 bytes; actions:
transmit
violated 0 packets, 0 bytes; actions:
drop
conformed 0000 bps, exceeded 0000 bps, violated 0000 bps
Hardware Counters:
class-map: CoPP-Catch-All (match-any)
Match: access-group name CoPP-Limit-and-Permit-Catch-All
Match: access-group name CoPP-Limit-and-Permit-Catch-All-IPv6
police :
496000 bps 15000 limit 15000 extended limit
Earl in slot 6 :
0 bytes
5 minute offered rate 0 bps
aggregate-forwarded 0 bytes action: transmit
exceeded 0 bytes action: transmit
aggregate-forward 0 bps exceed 0 bps
Software Counters:
Class-map: CoPP-Catch-All (match-any)
25991884 packets, 2564942791 bytes
5 minute offered rate 3000 bps, drop rate 0000 bps
Match: access-group name CoPP-Limit-and-Permit-Catch-All
25706499 packets, 2531503884 bytes
5 minute rate 3000 bps
Match: access-group name CoPP-Limit-and-Permit-Catch-All-IPv6
285385 packets, 33438907 bytes
5 minute rate 0 bps
police:
cir 500000 bps, bc 15625 bytes, be 15625 bytes
conformed 25990453 packets, 2564284549 bytes; actions:
transmit
exceeded 1375 packets, 578866 bytes; actions:
transmit
violated 56 packets, 79376 bytes; actions:
drop
conformed 3000 bps, exceeded 0000 bps, violated 0000 bps
Hardware Counters:
class-map: CoPP-Limit-and-Permit-Management-Plane-SYN (match-any)
Match: access-group name CoPP-Limit-and-Permit-TACACS-SYN
Match: access-group name CoPP-Limit-and-Permit-TELNET-SSH-SYN
police :
248000 bps 3000 limit 3000 extended limit
Earl in slot 6 :
0 bytes
5 minute offered rate 0 bps
aggregate-forwarded 0 bytes action: transmit
exceeded 0 bytes action: transmit
aggregate-forward 0 bps exceed 0 bps
Software Counters:
Class-map: CoPP-Limit-and-Permit-Management-Plane-SYN (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: access-group name CoPP-Limit-and-Permit-TACACS-SYN
0 packets, 0 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-TELNET-SSH-SYN
0 packets, 0 bytes
5 minute rate 0 bps
police:
cir 250000 bps, bc 3125 bytes, be 3125 bytes
conformed 0 packets, 0 bytes; actions:
transmit
exceeded 0 packets, 0 bytes; actions:
transmit
violated 0 packets, 0 bytes; actions:
drop
conformed 0000 bps, exceeded 0000 bps, violated 0000 bps
Hardware Counters:
class-map: CoPP-Limit-and-Permit-Critical-SYN (match-any)
Match: access-group name CoPP-Limit-and-Permit-BGP-SYN
Match: access-group name CoPP-Limit-and-Permit-BGPv6-SYN
Match: access-group name CoPP-Limit-and-Permit-LDP-SYN
police :
248000 bps 7000 limit 7000 extended limit
Earl in slot 6 :
0 bytes
5 minute offered rate 0 bps
aggregate-forwarded 0 bytes action: transmit
exceeded 0 bytes action: transmit
aggregate-forward 0 bps exceed 0 bps
Software Counters:
Class-map: CoPP-Limit-and-Permit-Critical-SYN (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: access-group name CoPP-Limit-and-Permit-BGP-SYN
0 packets, 0 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-BGPv6-SYN
0 packets, 0 bytes
5 minute rate 0 bps
Match: access-group name CoPP-Limit-and-Permit-LDP-SYN
0 packets, 0 bytes
5 minute rate 0 bps
police:
cir 250000 bps, bc 7812 bytes, be 7812 bytes
conformed 0 packets, 0 bytes; actions:
transmit
exceeded 0 packets, 0 bytes; actions:
transmit
violated 0 packets, 0 bytes; actions:
drop
conformed 0000 bps, exceeded 0000 bps, violated 0000 bps
Below we can see the internal gigabit connection to the RSP and how much traffic we are sending/receiving to/from it, and dropping. When a PFC or DFC decides to punt a packet it instructs the ingress line card to send the packet through the switch fabric to the fabric or bus interface on the supervisor/RSP. The fabric or bus interface forwards the packet to the packet ASIC on the RSP module. The packet ASIC forwards the packet to either the SP or RP CPU.
Each CPU has a separate In-band Interface Channel (IBC) with two input queues. The high priority queue (queue 0) receives packets with the data bus CoS value from 4 to 7 and low priority queue (queue 1) receives packets with the databus CoS value from 0 to 3. See the two RX queues below:
abr1#show ibc
Interface information:
Interface IBC0/0(idb 0x1D1CBF88)
5 minute rx rate 147000 bits/sec, 217 packets/sec
5 minute tx rate 281000 bits/sec, 202 packets/sec
1121195907 packets input, 92510091102 bytes
108882982 broadcasts received
1050940422 packets output, 173920811941 bytes
117764347 broadcasts sent
0 Bridge Packet loopback drops
511989670 Packets CEF Switched, 21 Packets Fast Switched
0 Packets SLB Switched, 0 Packets CWAN Switched
Label switched pkts dropped: 10 Pkts dropped during dma: 130
Invalid pkts dropped: 0 Pkts dropped(not cwan consumed): 0
Pkts marked to drop by VLAN clients: 0
IPSEC pkts: 1543
Xconnect pkts processed: 0, dropped: 0
Xconnect pkt reflection drops: 0
Total paks copied for process level 0
Total short paks sent in route cache 161251443
Total throttle drops 46 Input queue drops 5316
total spd packets classified (198955545 low, 345036684 medium, 52655209 high)
total spd packets dropped (129 low, 1 medium, 0 high)
spd prio pkts allowed in due to selective throttling (0 med, 0 high)
IBC resets = 1; last at 00:07:55.471 BST Fri Jun 6 2014
Driver Level Counters: (Cumulative, Zeroed only at Reset)
Frames Bytes
Rx(0) 55849743 2464316430
Rx(1) 1065352741 2142727136
Tx(0) 1050946996 1544836869
abr1#show mls statistics module 6
Statistics for Earl in Module 6
L2 Forwarding Engine
Total packets Switched : 2205950654529
L3 Forwarding Engine
Total packets Processed : 2205866081585 @ 1260305 pps
Total packets L3 Switched : 1974895372912 @ 1167585 pps
Total Packets Bridged : 21411016783
Total Packets FIB Switched : 1974895372912
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 0
Total Mcast Packets Switched/Routed : 172466214
Total ip packets with TOS changed : 28805093757
Total ip packets with COS changed : 33136415156
Total non ip packets COS changed : 73930452703
Total packets dropped by ACL : 16567242
Total packets dropped by Policing : 126602138
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 0
Short IP packets received : 0
IP header checksum errors : 0
No-route packet drops : 991111396
TTL failures : 23942682
MTU failures : 654618
When the IBC controller receives the packet, it copies the packet into IOS input/output memory and raises a Network Input/Output (NetIO) interrupt to the relivant CPU (RP or SP). More info here.
Previous page: BCP38 and Access Layer Filtering
Next page: Embedded Packet Capture (EPC)