Date created: Thursday, May 26, 2016 12:15:25 PM. Last modified: Friday, December 8, 2017 10:17:27 AM
BFD
References:
https://tools.ietf.org/html/rfc5880#section-6
http://www.cisco.com/en/US/technologies/tk648/tk365/tk480/technologies_white_paper0900aecd80244005.html
http://www.cisco.com/c/en/us/td/docs/switches/metro/me3600x_3800x/software/release/15-4_3_S/chassis/configuration/guide/3600x_24cxscg/swbfd.html
http://www.cisco.com/c/en/us/td/docs/switches/metro/me3600x_3800x/software/release/15-2_2_S/chassis/configuration/guide/3600x_24cxscg/swbfd.html
https://supportforums.cisco.com/document/144626/bfd-support-cisco-asr9000
http://www.cisco.com/c/en/us/td/docs/switches/metro/me3600x_3800x/software/release/15-3_3_S/configuration/guide/3800x3600xscg/swiprout.html#97750
Contents:
BFD Operations
BFD Examples
BFD "No Echo Mode" Support
BFD Operations
BFD (Bidirectional Forwarding Detection) operates asynchronously between two peers (BFD supports Asynchronous mode and Demand mode, Demand mode is not supported by Cisco devices). In BFD version 0 two peers send BFD control packets every N ms and after multiplier*N ms of not receiving a control packet from a remote peer, the local peer declares the link and session as down. If the local peer is slow to respond to BFD packets sent by the remote peer, but the remote peer is fast to respond to BFD packets sent by the local peer, the remote peer may decide the link is down whilst the local peer still thinks it is up, their opinion of the link state will be different, the remote will stop sending BFD packets if the link is shut down and eventually the local peer will then come to the same opinion that the link is down (due to no BFD responses). This will have happened at different rates (asynchronously) between the peers and so there is potential for packet loss during this brief period of differing opinions whilst the local peer “catches up” (even though the link is actually up).
With BFD version 0 the packets sent every N ms contain the control information for the BFD session (they are BFD control packets) so all packets need to be punted to the CPU to check the received session details in the received BFD packet match what is configured locally. This causes issues such as high CPU usage having to punt the BFD packets to the CPU and having to wait for interrupts to be processed, this means the BFD delay between peers will vary as CPU utilisation varies. A burst of high CPU usage could cause a BFD session to time out (not responding within the window of multiplied*N) due to the punted packets being processed slowly, causing a link to be signalled as down to the IGP and causing a re-convergence even though the link is fully operational.
BFD version 1 supports BFD echo mode (version 1 is the default version on all modern IOS/IOS-XE/IOS-XR versions, and will negotiate down to version 0 if required). In BFD echo mode two packet types are used. BFD echo packets which are sent every N ms and BFD control packets are sent at a slower rate (BFD “slow-timer”). In echo mode the local peer sends the BFD echo packet to the remote peer, the remote peer loops the packet internally and sends it back the way it came in (this is a hardware accelerated process, so the devices must have support for BFD hardware offload from the CPU) and the local peer receives the BFD echo packet it originally sent out. The remote peer is also sending out BFD echo packets that the local peer is looping back, without having to wait for CPU time. BFD control packets are being sent every BFD “slow-timer” interval and these are processed by the CPU of each peer. Now that that the BFD echo packets are being supported in hardware the control packets are not required so regularly, only to initiate the session and agree on parameters, the frequency of control packets can be lowered to save CPU utilisation. BFD echo mode is the default on all new IOS/IOS-XE/IOS-XR version and it must be disabled on platforms that don't support BFD echo mode in hardware using "no bfd echo" under the interface configuration. If disabled the device is still sending separate control and echo packets but they are all being processed by the CPU.
BFD Examples
Below is example output of a BFD session running between an ASR920 and ASR9001 which both support BFD echo mode and hardware acceleration. The BFD echo packet interval is configured as 50ms with a 3x timeout multiplier. The BFD slow-timer for BFD control packets is set to 5000ms. On the ASR9001 it can be seen that the average latency for BFD echo packets is 52ms (so fairly stable, due to the hardware support) but the average latency for async (control) packets is 2282ms on Tx and 4883ms on Rx, due to the CPU scheduling and load fluctuations. Also it shows "Echo detection time: 150 ms(50 ms*3), async detection time: 15 s(5 s*3)", the BFD echo packet interval set on both peers is 50ms and the timeout multiplier set is 3 so 150ms is the expected BFD timeout rate. For control packets in async mode on IOS-XR the interval is hard coded to 2s however on the ASR920 I have raised it to 5s so the ASR9001 has lowered to 5s too. Each remote peer uses the remotely configured multiplier as the local multiplier for the control packets, so both peers are also using 3*15s timeout for control packets. Note on the ASR920 it states "Session Host: Hardware" so hardware acceleration is present for the BFD echo packets:
RP/0/RSP0/CPU0:ASR9001#show bfd ipv4 session interface te0/0/1/2 detail
I/f: TenGigE0/0/1/2, Location: 0/0/CPU0
Dest: 10.0.2.1
Src: 10.0.2.2
State: UP for 0d:0h:5m:16s, number of times UP: 1
Session type: PR/V4/SH
Received parameters:
Version: 1, desired tx interval: 5 s, required rx interval: 5 s
Required echo rx interval: 50 ms, multiplier: 3, diag: None
My discr: 1, your discr: 2148073642, state UP, D/F/P/C/A: 0/0/0/1/0
Transmitted parameters:
Version: 1, desired tx interval: 2 s, required rx interval: 2 s
Required echo rx interval: 1 ms, multiplier: 3, diag: None
My discr: 2148073642, your discr: 1, state UP, D/F/P/C/A: 0/0/0/1/0
Timer Values:
Local negotiated async tx interval: 5 s
Remote negotiated async tx interval: 5 s
Desired echo tx interval: 50 ms, local negotiated echo tx interval: 50 ms
Echo detection time: 150 ms(50 ms*3), async detection time: 15 s(5 s*3)
Local Stats:
Intervals between async packets:
Tx: Number of intervals=100, min=1 ms, max=4960 ms, avg=2282 ms
Last packet transmitted 1441 ms ago
Rx: Number of intervals=64, min=10 ms, max=8755 ms, avg=4883 ms
Last packet received 8689 ms ago
Intervals between echo packets:
Tx: Number of intervals=100, min=51 ms, max=54 ms, avg=52 ms
Last packet transmitted 26 ms ago
Rx: Number of intervals=100, min=50 ms, max=55 ms, avg=52 ms
Last packet received 24 ms ago
Latency of echo packets (time between tx and rx):
Number of packets: 100, min=1 ms, max=5 ms, avg=3070 us
Session owner information:
Desired Adjusted
Client Interval Multiplier Interval Multiplier
-------------------- --------------------- ---------------------
ospf-1 50 ms 3 2 s 3
ASR920#show bfd neighbors ipv4 10.0.2.2 details
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.0.2.2 1/2148073642 Up Up Te0/0/2
Session state is UP and using echo function with 50 ms interval.
Session Host: Hardware - session negotiated with platform adjusted timer values.
MinTxInt - configured: 50000 adjusted: 5000000
OurAddr: 10.0.2.1
Handle: 1
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 5000000, MinRxInt: 5000000, Multiplier: 3
Received MinRxInt: 2000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 5000(0)
Rx Count: 8051154
Tx Count: 16865426
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: CEF OSPF FRR
Uptime: 00:05:02
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 1
Multiplier: 3 - Length: 24
My Discr.: 2148073642 - Your Discr.: 1
Min tx interval: 2000000 - Min rx interval: 2000000
Min Echo interval: 1000
! ASR9001
interface TenGigE0/0/1/2
description ASR920:Te0/0/2
ipv4 address 10.0.2.2 255.255.255.252
!
bfd
interface TenGigE0/0/1/2
echo
!
router ospf 1
router-id 1.0.0.3
area 0
interface Loopback0
!
interface TenGigE0/0/1/2
bfd minimum-interval 50
bfd fast-detect
bfd multiplier 3
cost 10
network point-to-point
! ASR920
bfd slow-timers 5000
!
interface TenGigabitEthernet0/0/2
description ASR9001:Te0/0/1/2
ip address 10.0.2.1 255.255.255.252
no ip redirects
ip ospf network point-to-point
ip ospf 1 area 0
bfd interval 50 min_rx 50 multiplier 3
!
router ospf 1
router-id 1.0.0.2
network 1.0.0.2 0.0.0.0 area 0
bfd all-interfaces
Below is output from a BFD session between an ME3600X and ME3800X which both don't support hardware offloading for the BFD echo packets. Note on both devices it states "Session Host: Software" verifying that hardware offload is not supported. Also note that the average latency for BFD packets is very high and there is wide range between min/max/avg times (due to the varying CPU utilisation and interrupt processing delay):
ME3800#show bfd neighbors interface te0/2 details
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.0.4.2 2/1 Up Up Te0/2
Session state is UP and using echo function with 50 ms interval.
Session Host: Software
OurAddr: 10.0.4.1
Handle: 2
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 5000000, MinRxInt: 5000000, Multiplier: 3
Received MinRxInt: 5000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 5000(4198653)
Rx Count: 72, Rx Interval (ms) min/max/avg: 1/4976/4323 last: 2348 ms ago
Tx Count: 74, Tx Interval (ms) min/max/avg: 1/4968/4217 last: 1436 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: CEF OSPF FRR
Uptime: 1d22h
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 1 - Your Discr.: 2
Min tx interval: 5000000 - Min rx interval: 5000000
Min Echo interval: 50000
ME3600#show bfd neighbors interface te0/2 details
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.0.4.1 1/2 Up Up Te0/2
Session state is UP and using echo function with 50 ms interval.
Session Host: Software
OurAddr: 10.0.4.2
Handle: 1
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 5000000, MinRxInt: 5000000, Multiplier: 3
Received MinRxInt: 5000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 5000(4202709)
Rx Count: 71, Rx Interval (ms) min/max/avg: 1/4968/4262 last: 4008 ms ago
Tx Count: 71, Tx Interval (ms) min/max/avg: 1/4976/4319 last: 76 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF CEF FRR
Uptime: 1d22h
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 2 - Your Discr.: 1
Min tx interval: 5000000 - Min rx interval: 5000000
Min Echo interval: 50000
In the below output (between the same ME3800X and ME3600X) the average latency on the BFD echo packets is 47ms on average. On the interface configuration the "no bfd echo" command has been added ("bfd echo" is the default, which gives the results above).
ME3800#show bfd neighbors interface Te0/2 details
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.0.4.2 2/1 Up Up Te0/2
Session state is UP and not using echo function.
Session Host: Software
OurAddr: 10.0.4.1
Handle: 2
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 50000, MinRxInt: 50000, Multiplier: 3
Received MinRxInt: 50000, Received Multiplier: 3
Holddown (hits): 118(0), Hello (hits): 50(4184460)
Rx Count: 3314443, Rx Interval (ms) min/max/avg: 1/72/47 last: 36 ms ago
Tx Count: 3310865, Tx Interval (ms) min/max/avg: 1/72/47 last: 40 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: CEF OSPF FRR
Uptime: 1d19h
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 1 - Your Discr.: 2
Min tx interval: 50000 - Min rx interval: 50000
Min Echo interval: 0
ME3600#show bfd neighbors interface te0/2 details
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.0.4.1 1/2 Up Up Te0/2
Session state is UP and not using echo function.
Session Host: Software
OurAddr: 10.0.4.2
Handle: 1
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 50000, MinRxInt: 50000, Multiplier: 3
Received MinRxInt: 50000, Received Multiplier: 3
Holddown (hits): 122(0), Hello (hits): 50(4188557)
Rx Count: 3310905, Rx Interval (ms) min/max/avg: 1/80/47 last: 28 ms ago
Tx Count: 3314483, Tx Interval (ms) min/max/avg: 1/72/47 last: 24 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF CEF FRR
Uptime: 1d19h
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 2 - Your Discr.: 1
Min tx interval: 50000 - Min rx interval: 50000
Min Echo interval: 0
!ME3800
bfd slow-timers 5000
!
interface TenGigabitEthernet0/2
description ME3600:Te0/2
no switchport
ip address 10.0.4.1 255.255.255.252
no ip redirects
ip ospf network point-to-point
ip ospf 1 area 0
bfd interval 50 min_rx 50 multiplier 3
no bfd echo
!
router ospf 1
router-id 1.0.0.4
network 1.0.0.4 0.0.0.0 area 0
bfd all-interfaces
!ME3600
bfd slow-timers 5000
!
interface TenGigabitEthernet0/2
description ME3800:Te0/2
no switchport
ip address 10.0.4.2 255.255.255.252
no ip redirects
ip ospf network point-to-point
ip ospf 1 area 0
bfd interval 50 min_rx 50 multiplier 3
no bfd echo
!
router ospf 1
router-id 1.0.0.5
network 1.0.0.4 0.0.0.0 area 0
bfd all-interfaces
The output directly above when compared to the output above that is confusing because when "bfd echo" is configured the average BFD echo packet latency between the two ME switches is much higher. With echo mode enabled the ME switches send echo packets from the BFD software layer (on the local peer) and the remote BFD neighbour responds to the echo packets through its fast-switching layer (hardware accelerated forwarding). This is because the echo packets are UDP packets and the source and destination IP are both the address of the sender, so the remote peer can "loop" the echo packet back to the sender in hardware (there is no need to CPU punt a standard UDP packet). This means the echo packets do not reach the remote BFD neighbour’s software layer and are reflected back over the remote forwarding path for fast failure detection. When the packets come back though they need to go back up to the local BFD software layer for processing again. For the local peer it has to originate the packets in software and check them upon return arrival in software. This is why the latency times are high with "bfd echo" (default) configured, because BFD hardware offload is not supported on these two example devices for echo mode.
On these two example switches, when "no bfd echo" is configured under the interface control packets are used to detect forwarding failures (echo packets are no longer sent). The BFD interval is used to exchange control packets at the rate echo packets would have been exchanged. In bfd no echo mode the configured BFD interval values are negotiated at the slow-timer rate and the BFD peers send control packets to each other at the negotiated BFD intervals. Each peer sends packets to the other and checks the packets they have received from their neighbour, they are not looped back and checked by the originating peer. No echo mode is support in hardware on the ME3600X/ME3800X/7600 devices. It's a strange design from Cisco because in echo mode the local peer originates packets and checks it's own packets when they are received again. In no echo mode the local peer originates packets but checks the packets the remote peer sent only. So in both case two CPU tasks which happen at the same rate but for some reason the Cisco docs only show non-echo mode as being supported on these devices.
The output below is between an ME3600X and 7606-S using a WS-X6704-10G line card without DFC (two devices with no hardware BFD echo mode acceleration). Both show "Session Host: Software" and the bfd configuration is "bfd interval 500 min_rx 500 multiplier 3" with BFD echo enabled by default. It can be seen that neither device has an average BFD interval of 500ms until after BFD echo is disabled on both device interfaces and then the average interval falls to just above 400ms (the BFD RFC states there can be a 25% variance in the arrival time of BFD packets to what is configured). That shows there is a significant performance increase due to the hardware acceleration for BFD non echo mode in these platforms. The opposite can be seen in the ASR920 and ASR9001 test devices used above, when switching to non-echo mode on those devices with hardware offload for echo mode, the BFD packet timers slip and because the CPU can't keep up:
ME3600X#show bfd neighbors ipv4 10.0.253.1 details
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.0.253.1 8/8070 Up Up Te0/1
Session state is UP and using echo function with 500 ms interval.
Session Host: Software
OurAddr: 10.0.253.2
Handle: 1
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 1000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 1000(1302976)
Rx Count: 1305954, Rx Interval (ms) min/max/avg: 1/1008/883 last: 680 ms ago
Tx Count: 1302978, Tx Interval (ms) min/max/avg: 1/1016/881 last: 932 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF CEF
Uptime: 1w6d
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 8070 - Your Discr.: 8
Min tx interval: 1000000 - Min rx interval: 1000000
Min Echo interval: 500000
7606-S#show bfd neighbors interface te3/1 details
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.0.253.2 8076/9 Up Up Te3/1
Session state is UP and using echo function with 500 ms interval.
Session Host: Software
OurAddr: 10.0.253.1
Handle: 41
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 1000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 1000(217)
Rx Count: 219, Rx Interval (ms) min/max/avg: 1/1000/873 last: 548 ms ago
Tx Count: 218, Tx Interval (ms) min/max/avg: 756/1000/877 last: 504 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF CEF
Uptime: 00:03:10
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 9 - Your Discr.: 8076
Min tx interval: 1000000 - Min rx interval: 1000000
Min Echo interval: 500000
ME3600X#conf t
ME3600X(config-if)#int te0/1
ME3600X(config-if)#no bfd echo
7606-S#conf t
7606-S(config)#int te3/1
7606-S(config-if)#no bfd echo
ME3600X#show bfd neighbors ipv4 10.0.253.1 details
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.0.253.1 9/8076 Up Up Te0/1
Session state is UP and not using echo function.
Session Host: Software
OurAddr: 10.0.253.2
Handle: 1
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 500000, MinRxInt: 500000, Multiplier: 3
Received MinRxInt: 500000, Received Multiplier: 3
Holddown (hits): 1324(0), Hello (hits): 500(2706)
Rx Count: 67, Rx Interval (ms) min/max/avg: 1/500/431 last: 176 ms ago
Tx Count: 68, Tx Interval (ms) min/max/avg: 1/512/423 last: 336 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF CEF
Uptime: 00:39:23
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 8076 - Your Discr.: 9
Min tx interval: 500000 - Min rx interval: 500000
Min Echo interval: 0
7606-S#show bfd neighbors interface te3/1 details
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.0.253.2 8076/9 Up Up Te3/1
Session state is UP and not using echo function.
Session Host: Software
OurAddr: 10.0.253.1
Handle: 41
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 500000, MinRxInt: 500000, Multiplier: 3
Received MinRxInt: 500000, Received Multiplier: 3
Holddown (hits): 1276(0), Hello (hits): 500(2704)
Rx Count: 50, Rx Interval (ms) min/max/avg: 1/512/408 last: 224 ms ago
Tx Count: 49, Tx Interval (ms) min/max/avg: 1/500/416 last: 144 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF CEF
Uptime: 00:39:14
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 9 - Your Discr.: 8076
Min tx interval: 500000 - Min rx interval: 500000
Min Echo interval: 0
Previous page: ASR9000 Load-Balancing
Next page: Cisco and Juniper IS-IS Interop