Date created: Wednesday, September 6, 2017 8:58:10 AM. Last modified: Wednesday, September 27, 2017 3:44:11 PM
Route Distinguisher Significance
Contents
Initial Config
Route Distinguisher Recap
RD Local Significance
RD Global Significance
Unique RDs and Unique RTs per VRF per device
Conclusion
Type 1 RDs are being used throughout which are comprised from a 4 byte admin subfield and 2 byte assigned number subfield.
This is the initial topology:
PE1#show run vrf definition CUST rd 1.1.1.1:1 ! address-family ipv4 route-target export 1:1 route-target import 1:1 exit-address-family ! interface Loopback0 ip address 10.0.0.1 255.255.255.255 ! interface Loopback1 vrf forwarding CUST ip address 2.2.2.1 255.255.255.255 ! interface FastEthernet0/0 description PE2-Fa0/0 ip address 10.0.1.1 255.255.255.252 ip ospf network point-to-point ip ospf 1 area 0 ! router ospf 1 router-id 10.0.0.1 network 10.0.0.1 0.0.0.0 area 0 ! router bgp 1 bgp router-id 10.0.0.1 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor 10.0.0.2 remote-as 1 neighbor 10.0.0.2 update-source Loopback0 ! address-family ipv4 exit-address-family ! address-family vpnv4 neighbor 10.0.0.2 activate neighbor 10.0.0.2 send-community both neighbor 10.0.0.2 next-hop-self exit-address-family ! address-family ipv4 vrf CUST redistribute connected exit-address-family ! PE2 vrf definition CUST rd 1.1.1.1:1 ! address-family ipv4 route-target export 1:2 route-target import 1:2 exit-address-family ! interface Loopback0 ip address 10.0.0.2 255.255.255.255 ! interface Loopback1 vrf forwarding CUST ip address 2.2.2.2 255.255.255.255 ! interface FastEthernet0/0 description PE1-Fa0/0 ip address 10.0.1.2 255.255.255.252 ip ospf network point-to-point ip ospf 1 area 0 ! router ospf 1 router-id 10.0.0.2 network 10.0.0.2 0.0.0.0 area 0 ! router bgp 1 bgp router-id 10.0.0.2 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor 10.0.0.1 remote-as 1 neighbor 10.0.0.1 update-source Loopback0 ! address-family ipv4 exit-address-family ! address-family vpnv4 neighbor 10.0.0.1 activate neighbor 10.0.0.1 send-community both neighbor 10.0.0.1 next-hop-self exit-address-family ! address-family ipv4 vrf CUST redistribute connected exit-address-family
When PE1 and PE2 have the same (or different) RDs but the RTs don’t match, it goes without saying that routes will not be imported into the the local VRF RIB. Below it can be seen that PE1 has not imported 2.2.2.2/32 from PE2 into the CUST VRF RIB, because the PE1 RT is 1:1 and PE2 RT is 1:2, also that PE1 isn't even keeping a copy of the route for this prefix in its VPNv4 table:
PE1#show run vrf CUST
Building configuration...
Current configuration : 313 bytes
vrf definition CUST
rd 1.1.1.1:1
!
address-family ipv4
route-target export 1:1
route-target import 1:1
exit-address-family
PE2#show run vrf CUST
Building configuration...
Current configuration : 313 bytes
vrf definition CUST
rd 1.1.1.1:1
!
address-family ipv4
route-target export 1:2
route-target import 1:2
exit-address-family
PE1#show bgp vpnv4 unicast all
BGP table version is 2, local router ID is 10.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1 (default for vrf CUST)
*> 2.2.2.1/32 0.0.0.0 0 32768 ?
PE2#conf t
Enter configuration commands, one per line. End with CNTL/Z.
PE2(config)#vrf definition CUST
PE2(config-vrf)#address-family ipv4
PE2(config-vrf-af)#no route-target both
PE2(config-vrf-af)#route-target both 1:1
PE1#show bgp vpnv4 unicast all
BGP table version is 4, local router ID is 10.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1 (default for vrf CUST)
*> 2.2.2.1/32 0.0.0.0 0 32768 ?
*>i 2.2.2.2/32 10.0.0.2 0 100 0 ?
The difference in the above now is that the RTs are the same so PE1 is "storing" the route in its VPNv4 table. Previously when the RTs were different PE2 was sending the route and PE1 was dropping it because no VRFs were configure with the same RT as an import RT. Side note: if the RT filter address-family was configured PE2 wouldn't have even sent the route, but in this example that AF is not configured for debugging and demo purposes.
PE1 and PE2 use the same RD for the CUST VRF 1.1.1.1:1. This means over on PE1 we see both the prefixes shown above sitting under the RD 1.1.1.1:1 in the VPNv4 table and only one copy of the prefix 1.1.1.1:1:2.2.2.2/32:
PE1#show bgp vpnv4 unicast rd 1.1.1.1:1
BGP table version is 4, local router ID is 10.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1 (default for vrf CUST)
*> 2.2.2.1/32 0.0.0.0 0 32768 ?
*>i 2.2.2.2/32 10.0.0.2 0 100 0 ?
PE1#show bgp vpnv4 unicast rd 1.1.1.1:1 2.2.2.2/32
BGP routing table entry for 1.1.1.1:1:2.2.2.2/32, version 4
Paths: (1 available, best #1, table CUST)
Not advertised to any peer
Refresh Epoch 1
Local
10.0.0.2 (metric 2) from 10.0.0.2 (10.0.0.2)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/16
So far it looks like the RD that PE2 uses for prefixes in the CUST VRF is truly locally significant. If we change it to be unique we get the following output from PE1:
PE2#conf t
PE2(config)#vrf definition CUST
PE2(config-vrf)#no rd 1.1.1.1:1
% "rd 1.1.1.1:1" for VRF CUST scheduled for deletion
PE2(config-vrf)#rd 1.1.1.1:2
PE2(config-vrf)#address-family ipv4
PE2(config-vrf-af)#route-target both 1:1
PE2(config-vrf-af)#exit
PE2(config-vrf)#exit
PE2(config)#router bgp 1
PE2(config-router)#address-family ipv4 vrf CUST
PE2(config-router-af)#redistribute connected
PE2#show run vrf CUST
Building configuration...
Current configuration : 219 bytes
vrf definition CUST
rd 1.1.1.1:2
!
address-family ipv4
route-target export 1:1
route-target import 1:1
exit-address-family
PE1#show bgp vpnv4 unicast all
BGP table version is 7, local router ID is 10.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1 (default for vrf CUST)
*> 2.2.2.1/32 0.0.0.0 0 32768 ?
*>i 2.2.2.2/32 10.0.0.2 0 100 0 ?
Route Distinguisher: 1.1.1.1:2
*>i 2.2.2.2/32 10.0.0.2 0 100 0 ?
The topology now looks like this:
Above we can see there are now two 2.2.2.2/32 prefix entries on PE1, one with each RD 1.1.1.1:1 and 1.1.1.1:2. Below we can see one is imported into the local vrf CUST because it has the same RT (1:1):
PE1#show bgp vpnv4 unicast vrf CUST 2.2.2.2/32
BGP routing table entry for 1.1.1.1:1:2.2.2.2/32, version 7
Paths: (1 available, best #1, table CUST)
Not advertised to any peer
Refresh Epoch 1
Local, imported path from 1.1.1.1:2:2.2.2.2/32
10.0.0.2 (metric 2) from 10.0.0.2 (10.0.0.2)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/17
However, the original copy of the prefix with RD 1.1.1.1:2 is kept in the VPNv4 table because it had an export RT that matched the import RT of a VRF configured on the local PE. Within the scope of MP-BGP and L3VPNs the RD helps to keep overlapping routes unique (as the name suggested it "distinguishe" one route from another).
Below it can be seen that PE1 now actually has two routes to 2.2.2.2/32 in the VPNv4 table because of this:
PE1#show bgp vpnv4 unicast all 2.2.2.2/32
BGP routing table entry for 1.1.1.1:1:2.2.2.2/32, version 7
Paths: (1 available, best #1, table CUST)
Not advertised to any peer
Refresh Epoch 1
Local, imported path from 1.1.1.1:2:2.2.2.2/32
10.0.0.2 (metric 2) from 10.0.0.2 (10.0.0.2)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/17
BGP routing table entry for 1.1.1.1:2:2.2.2.2/32, version 6
Paths: (1 available, best #1, no table)
Not advertised to any peer
Refresh Epoch 1
Local
10.0.0.2 (metric 2) from 10.0.0.2 (10.0.0.2)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/17
Initially this seems like no problem however the local RD configured on PE2 has had some impact on PE1. One minor port is that with matching RDs PE1 has only one copy of the route from PE2:
PE1#show bgp vpnv4 unicast all summary BGP router identifier 10.0.0.1, local AS number 1 BGP table version is 11, main routing table version 11 2 network entries using 336 bytes of memory 2 path entries using 128 bytes of memory 2/2 BGP path/bestpath attribute entries using 288 bytes of memory 1 BGP extended community entries using 24 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 776 total bytes of memory BGP activity 5/3 prefixes, 5/3 paths, scan interval 60 secs
With unique RDs PE1 has two copies of the route and uses 50% extra memory, not much of a different with one prefix but with thousands of routes per VRF (e.g. Internet in a VRF) this could have some impact:
PE1#show bgp vpnv4 unicast all summary BGP router identifier 10.0.0.1, local AS number 1 BGP table version is 7, main routing table version 7 3 network entries using 504 bytes of memory 3 path entries using 192 bytes of memory 2/2 BGP path/bestpath attribute entries using 288 bytes of memory 1 BGP extended community entries using 24 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 1008 total bytes of memory BGP activity 4/1 prefixes, 4/1 paths, scan interval 60 secs
Now PE3 is added to the toplogy which also advertises 2.2.2.2/32 in the same CUST VRF with the same RD and Route Reflector 1 (RR1) is also added to the topology. All all iBGP peerings are removed and only configured to RR1 from each PE, another global impact from RDs can be observed now.
The topology now looks like this:
All PEs have the same config now with regards to BGP (any differences are as expected, loopback IPs and BGP/OSPF router IDs etc.):
PE3#show run vrf definition CUST rd 1.1.1.1:2 ! address-family ipv4 route-target export 1:1 route-target import 1:1 exit-address-family ! interface Loopback0 ip address 10.0.0.3 255.255.255.255 ! interface Loopback1 vrf forwarding CUST ip address 2.2.2.2 255.255.255.255 ! interface FastEthernet0/0 description PE2-Fa0/1 ip address 10.0.2.2 255.255.255.252 ip ospf network point-to-point ip ospf 1 area 0 ! interface FastEthernet0/1 description PE1-Fa0/1 ip address 10.0.3.2 255.255.255.252 ip ospf network point-to-point ip ospf 1 area 0 !
interface FastEthernet1/0
description RR1-Fa1/0
ip address 10.0.4.1 255.255.255.252
ip ospf network point-to-point
ip ospf 1 area 0
! router ospf 1 router-id 10.0.0.3 network 10.0.0.3 0.0.0.0 area 0 ! router bgp 1 bgp router-id 10.0.0.3 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor 10.0.0.4 remote-as 1 neighbor 10.0.0.4 description RR1 neighbor 10.0.0.4 update-source Loopback0 ! address-family ipv4 exit-address-family ! address-family vpnv4 neighbor 10.0.0.4 activate neighbor 10.0.0.4 send-community extended neighbor 10.0.0.4 next-hop-self exit-address-family ! address-family ipv4 vrf CUST redistribute connected exit-address-family ! RR1#show run interface Loopback0 ip address 10.0.0.4 255.255.255.255 ! interface FastEthernet1/0 description PE3-Fa1/0 ip address 10.0.4.2 255.255.255.252 ip ospf network point-to-point ip ospf 1 area 0 ! router ospf 1 router-id 10.0.0.4 network 10.0.0.4 0.0.0.0 area 0 ! router bgp 1 bgp router-id 10.0.0.4 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor iBGP-RR-Clients peer-group neighbor iBGP-RR-Clients remote-as 1 neighbor iBGP-RR-Clients update-source Loopback0 neighbor 10.0.0.1 peer-group iBGP-RR-Clients neighbor 10.0.0.1 description PE1 neighbor 10.0.0.2 peer-group iBGP-RR-Clients neighbor 10.0.0.2 description PE2 neighbor 10.0.0.3 peer-group iBGP-RR-Clients neighbor 10.0.0.3 description PE3 ! address-family ipv4 exit-address-family ! address-family vpnv4 neighbor iBGP-RR-Clients send-community extended neighbor iBGP-RR-Clients route-reflector-client neighbor 10.0.0.1 activate neighbor 10.0.0.2 activate neighbor 10.0.0.3 activate exit-address-family
At this point PE3 has the same RD as PE2 for VRF CUST and is advertising the same 2.2.2.2/32 prefix (this could be from a dual-homes CE for example). Below it can be seen that RR1 has one prefix from each PE, also it has two paths to the same route 1.1.1.1:2:2.2.2.2/32 via 10.0.0.2 and 10.0.0.3:
RR1#show bgp vpnv4 unicast all summary
BGP router identifier 10.0.0.4, local AS number 1
BGP table version is 6, main routing table version 6
2 network entries using 336 bytes of memory
3 path entries using 192 bytes of memory
1/1 BGP path/bestpath attribute entries using 144 bytes of memory
1 BGP extended community entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 696 total bytes of memory
BGP activity 3/1 prefixes, 4/1 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.1 4 1 13 16 6 0 0 00:08:02 1
10.0.0.2 4 1 13 18 6 0 0 00:08:01 1
10.0.0.3 4 1 15 27 6 0 0 00:08:02 1
RR1#show bgp vpnv4 unicast all
BGP table version is 6, local router ID is 10.0.0.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
Route Distinguisher: 1.1.1.1:2
*>i 2.2.2.2/32 10.0.0.3 0 100 0 ?
* i 10.0.0.2 0 100 0 ?
This is to be expected, because the RD is the same for the 2.2.2.2/32 announcement from PE2 and PE3, RR1 seems them as two paths to the same prefix, the problem is that now both paths compete on RR1 to become the BGP “best path” as in its current configuration (no use of BGP Add-Path/advertise best external etc) RR1 will only advertise the best-path to other PEs:
RR1#show bgp vpnv4 unicast rd 1.1.1.1:2 2.2.2.2/32
BGP routing table entry for 1.1.1.1:2:2.2.2.2/32, version 6
Paths: (2 available, best #1, no table)
Advertised to update-groups:
1
Refresh Epoch 1
Local, (Received from a RR-client)
10.0.0.3 (metric 2) from 10.0.0.3 (10.0.0.3)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/17
Refresh Epoch 1
Local, (Received from a RR-client)
10.0.0.2 (metric 3) from 10.0.0.2 (10.0.0.2)
Origin incomplete, metric 0, localpref 100, valid, internal
Extended Community: RT:1:1
mpls labels in/out nolabel/16
RR1#show bgp vpnv4 unicast all update-group 1
…
Has 3 members:
10.0.0.1 10.0.0.2 10.0.0.3
Over on PE1 only the route via PE3 is received, and it can be seen there are two copies. The original route with RD 1.1.1.1:2 and the route imported into the local 1.1.1.1:1 space because of the matching RT 1:1:
PE1#show bgp vpnv4 unicast all
BGP table version is 23, local router ID is 10.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1 (default for vrf CUST)
*> 2.2.2.1/32 0.0.0.0 0 32768 ?
*>i 2.2.2.2/32 10.0.0.3 0 100 0 ?
Route Distinguisher: 1.1.1.1:2
*>i 2.2.2.2/32 10.0.0.3 0 100 0 ?
PE1#show bgp vpnv4 unicast all 2.2.2.2/32
BGP routing table entry for 1.1.1.1:1:2.2.2.2/32, version 23
Paths: (1 available, best #1, table CUST)
Not advertised to any peer
Refresh Epoch 1
Local, imported path from 1.1.1.1:2:2.2.2.2/32
10.0.0.3 (metric 2) from 10.0.0.4 (10.0.0.4)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
Originator: 10.0.0.3, Cluster list: 10.0.0.4
mpls labels in/out nolabel/17
BGP routing table entry for 1.1.1.1:2:2.2.2.2/32, version 22
Paths: (1 available, best #1, no table)
Not advertised to any peer
Refresh Epoch 1
Local
10.0.0.3 (metric 2) from 10.0.0.4 (10.0.0.4)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
Originator: 10.0.0.3, Cluster list: 10.0.0.4
mpls labels in/out nolabel/17
The above has shown that the RD on PE3 has had an impact on which route is advertised to PE1 and that only one of two available routes were advertised, in the event of a failure between PE3 and the hypothetical dual-homed CE advertising 2.2.2.2./32 into AS1, PE1 isn’t aware that it could still send traffic towards PE2.
Adjusting the RD on PE3 to be unique (1.1.1.1:3) will allow PE1 to receive both paths as shown below and choose for its self which is the best path from its own perspective of the network:
PE3#conf t
PE3(config)#vrf def CUST
PE3(config-vrf)#no rd 1.1.1.1:2
% "rd 1.1.1.1:2" for VRF CUST scheduled for deletion
PE3(config-vrf)# rd 1.1.1.1:3
PE3(config-vrf)#address-family ipv4
PE3(config-vrf-af)#route-target both 1:1
PE3(config-vrf-af)#exit
PE3(config-vrf)#exit
PE3(config)#router bgp 1
PE3(config-router)#address-family ipv4 vrf CUST
PE3(config-router-af)#redistribute connected
PE1#show bgp vpnv4 unicast al
BGP table version is 26, local router ID is 10.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1 (default for vrf CUST)
*> 2.2.2.1/32 0.0.0.0 0 32768 ?
* i 2.2.2.2/32 10.0.0.3 0 100 0 ?
*>i 10.0.0.2 0 100 0 ?
Route Distinguisher: 1.1.1.1:2
*>i 2.2.2.2/32 10.0.0.2 0 100 0 ?
Route Distinguisher: 1.1.1.1:3
*>i 2.2.2.2/32 10.0.0.3 0 100 0 ?
PE1#show bgp vpnv4 unicast all 2.2.2.2/32
BGP routing table entry for 1.1.1.1:1:2.2.2.2/32, version 25
Paths: (2 available, best #2, table CUST)
Not advertised to any peer
Refresh Epoch 1
Local, imported path from 1.1.1.1:3:2.2.2.2/32
10.0.0.3 (metric 2) from 10.0.0.4 (10.0.0.4)
Origin incomplete, metric 0, localpref 100, valid, internal
Extended Community: RT:1:1
Originator: 10.0.0.3, Cluster list: 10.0.0.4
mpls labels in/out nolabel/16
Refresh Epoch 1
Local, imported path from 1.1.1.1:2:2.2.2.2/32
10.0.0.2 (metric 2) from 10.0.0.4 (10.0.0.4)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
Originator: 10.0.0.2, Cluster list: 10.0.0.4
mpls labels in/out nolabel/16
BGP routing table entry for 1.1.1.1:2:2.2.2.2/32, version 24
Paths: (1 available, best #1, no table)
Not advertised to any peer
Refresh Epoch 1
Local
10.0.0.2 (metric 2) from 10.0.0.4 (10.0.0.4)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
Originator: 10.0.0.2, Cluster list: 10.0.0.4
mpls labels in/out nolabel/16
BGP routing table entry for 1.1.1.1:3:2.2.2.2/32, version 26
Paths: (1 available, best #1, no table)
Not advertised to any peer
Refresh Epoch 1
Local
10.0.0.3 (metric 2) from 10.0.0.4 (10.0.0.4)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
Originator: 10.0.0.3, Cluster list: 10.0.0.4
mpls labels in/out nolabel/16
Above it can be seen that PE1 has four copies of a path to prefix 2.2.2.2/32, one is 1.1.1.1:2:2.2.2.2/32 and one is 1.1.1.1:3:2.2.2.2/32, which are the two copies RR1 send on behalf of PE2 and PE3 respectively. PE1 has kept them both in its VPNv4 table because they have an export RT (1:1) that matches the import RT of a locally configured VRF (CUST). The other two paths are for prefix 1.1.1.1:1:2.2.2.2/32 which are the copies imported into VRF CUST that PE1 performs its BGP best-path selection process against, and will ultimately offer up to the FIB.
The topology now looks like this:
It can be seen now that not only is PE2 preferred but also the path via PE3 is available as a backup. Below in the RR1 output it can be seen that there are two unique routes, 1.1.1.1:2:2.2.2.2/32 and 1.1.1.1:3:2.2.2.2/32 which are not compared using the BGP best path algorithm as their unique RDs makes them both unique paths:
RR1#show bgp vpnv4 unicast all 2.2.2.2/32
BGP routing table entry for 1.1.1.1:2:2.2.2.2/32, version 7
Paths: (1 available, best #1, no table)
Advertised to update-groups:
1
Refresh Epoch 1
Local, (Received from a RR-client)
10.0.0.2 (metric 3) from 10.0.0.2 (10.0.0.2)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/16
BGP routing table entry for 1.1.1.1:3:2.2.2.2/32, version 8
Paths: (1 available, best #1, no table)
Advertised to update-groups:
1
Refresh Epoch 1
Local, (Received from a RR-client)
10.0.0.3 (metric 2) from 10.0.0.3 (10.0.0.3)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/16
RR1#show bgp vpnv4 unicast rd 1.1.1.1:2 2.2.2.2/32
BGP routing table entry for 1.1.1.1:2:2.2.2.2/32, version 7
Paths: (1 available, best #1, no table)
Advertised to update-groups:
1
Refresh Epoch 1
Local, (Received from a RR-client)
10.0.0.2 (metric 3) from 10.0.0.2 (10.0.0.2)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/16
RR1#show bgp vpnv4 unicast rd 1.1.1.1:3 2.2.2.2/32
BGP routing table entry for 1.1.1.1:3:2.2.2.2/32, version 8
Paths: (1 available, best #1, no table)
Advertised to update-groups:
1
Refresh Epoch 1
Local, (Received from a RR-client)
10.0.0.3 (metric 2) from 10.0.0.3 (10.0.0.3)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/16
For a typical service provider network this is actually everyday stuff. Many SP's use the loobpack0 IP of a PE router to keep RDs unique and ensure that multiple paths exists within the iBGP domain to a multi-homed device or location. This is often extended with the uses of BGP-ORR (Optimal Route Reflection) or BGP-AP (Add-Path), but those are separate subjects out of scope here.
Unique RDs and Unique RTs per VRF per device
In the above example it was shown that RDs need to be unique, not just for backup path purposes but for optimal best path selection purposes too, and that the RD configured locally on a PE is affecting traffic flow around the network by affecting the aforementioned best path selection process.
Below ASBR1 has been added to the topology with a new VRF called ADMIN advertising the prefix 2.2.2.3/32. The VRF CUST is not present on ABSR1 so the same RD has been used for the ADMIN VRF on ASBR1 as was used for the CUST VRF on the PE2 router, 1.1.1.1:2. The ADMIN VRF has been added to PE3 but with a unique RD and RT and prefix, 1.1.1.1:4:2.2.2.4/32. The output below shows that PE3 and ABR1 have learnt each other’s prefix within the ADMIN VRF.
The topology now looks like this:
It can also be seen in the output below that RR1 has received the prefix 2.2.2.3/32 from ASBR1 and it has the same RD (1.1.1.1:2) as the PE2 route 2.2.2.2/32, however PE2 does not have this prefix in its VPNv4 table (no matching RT 1:2). Also 2.2.2.4/32 is received from PE3 on RR1, but again not by PE1 or PE2 as there is no matching RT (1:2):
RR1#show bgp vpnv4 unicast all
BGP table version is 10, local router ID is 10.0.0.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
Route Distinguisher: 1.1.1.1:2
*>i 2.2.2.2/32 10.0.0.2 0 100 0 ?
*>i 2.2.2.3/32 10.0.0.5 0 100 0 ?
Route Distinguisher: 1.1.1.1:3
*>i 2.2.2.2/32 10.0.0.3 0 100 0 ?
Route Distinguisher: 1.1.1.1:4
*>i 2.2.2.4/32 10.0.0.3 0 100 0 ?
PE2#show bgp vpnv4 unicast all
BGP table version is 29, local router ID is 10.0.0.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
Route Distinguisher: 1.1.1.1:2 (default for vrf CUST)
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
* i 2.2.2.2/32 10.0.0.3 0 100 0 ?
*> 0.0.0.0 0 32768 ?
Route Distinguisher: 1.1.1.1:3
*>i 2.2.2.2/32 10.0.0.3 0 100 0 ?
ASBR1#show bgp vpnv4 unicast all
BGP table version is 4, local router ID is 10.0.0.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:2 (default for vrf ADMIN)
*> 2.2.2.3/32 0.0.0.0 0 32768 ?
*>i 2.2.2.4/32 10.0.0.3 0 100 0 ?
Route Distinguisher: 1.1.1.1:4
*>i 2.2.2.4/32 10.0.0.3 0 100 0 ?
PE3#show bgp vpnv4 unicast all
BGP table version is 14, local router ID is 10.0.0.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
Route Distinguisher: 1.1.1.1:2
*>i 2.2.2.2/32 10.0.0.2 0 100 0 ?
*>i 2.2.2.3/32 10.0.0.5 0 100 0 ?
Route Distinguisher: 1.1.1.1:3 (default for vrf CUST)
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
* i 2.2.2.2/32 10.0.0.2 0 100 0 ?
*> 0.0.0.0 0 32768 ?
Route Distinguisher: 1.1.1.1:4 (default for vrf ADMIN)
*>i 2.2.2.3/32 10.0.0.5 0 100 0 ?
*> 2.2.2.4/32 0.0.0.0 0 32768 ?
At this point in time based on the output above it seems that there are no problems. Unique RDs are configured on all PEs with the CUST VRF (PE1/2/3) and a single globally unique RT is used. Traffic within the CUST VRF is working as desired (PE1/2/3 have the 2.2.2.1/32 prefix and PE1 prefers the PE2 route to the 2.2.2.2/32 prefix with PE3 as a backup path). Unique RDs are also used on all PEs configured with the ADMIN VRF (PE3 and ASBR1). Again, both PEs have 2.2.2.2/32 and 2.2.2.3/32 so everything here seems to be working as desired. The difference is that ASBR1 is using the exact same RD for ADMIN and CUST VRF, and some people think that RDs are only locally significant...
Within the CUST VRF the prefix 2.2.2.3/32 has been allocated and it is reachable via 2.2.2.2/32 (a LAN subnet hanging off of a fictional dual-homed site for example). Below a static route is added to PE2 to accommodate this. There should be no problem with 2.2.2.3/32 being allocated, it does overlap with the prefix ASBR1 advertises within the ADMIN L3 VPN but it’s a separate L3 VPN right, overlapping prefixes are no problem surely? That’s what VRFs are for yeah man?
PE2(config)#ip route vrf CUST 2.2.2.3 255.255.255.255 2.2.2.1
PE2(config)#router bgp 1
PE2(config-router)#address-family ipv4 vrf CUST
PE2(config-router-af)#redistribute static
PE2#show bgp vpnv4 unicast all
BGP table version is 30, local router ID is 10.0.0.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
Route Distinguisher: 1.1.1.1:2 (default for vrf CUST)
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
* i 2.2.2.2/32 10.0.0.3 0 100 0 ?
*> 0.0.0.0 0 32768 ?
*> 2.2.2.3/32 2.2.2.1 0 32768 ?
Route Distinguisher: 1.1.1.1:3
*>i 2.2.2.2/32 10.0.0.3 0 100 0 ?
The output above from PE looks OK. However in the output below 2.2.2.2/32 is now being received by RR1 with the same RD (1.1.1.1:2) from PE2 (with the CUST RT) and ASBR1 (with the ADMIN RT) which is now the exact same prefix within the VPNv4 table (1.1.1.1:2:2.2.2.2/32) and thus eligible for BGP best path comparison. Below it can be seen that 1.1.1.1:2:2.2.2.2/32 is preferred via 10.0.0.2 (PE2) on RR1:
RR1#show bgp vpnv4 unicast all
BGP table version is 11, local router ID is 10.0.0.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
Route Distinguisher: 1.1.1.1:2
*>i 2.2.2.2/32 10.0.0.2 0 100 0 ?
*>i 2.2.2.3/32 10.0.0.2 0 100 0 ?
* i 10.0.0.5 0 100 0 ?
Route Distinguisher: 1.1.1.1:3
*>i 2.2.2.2/32 10.0.0.3 0 100 0 ?
Route Distinguisher: 1.1.1.1:4
*>i 2.2.2.4/32 10.0.0.3 0 100 0 ?
Everything looks normal on ASBR1:
ASBR1#show bgp vpnv4 unicast all
BGP table version is 4, local router ID is 10.0.0.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:2 (default for vrf ADMIN)
*> 2.2.2.3/32 0.0.0.0 0 32768 ?
*>i 2.2.2.4/32 10.0.0.3 0 100 0 ?
Route Distinguisher: 1.1.1.1:4
*>i 2.2.2.4/32 10.0.0.3 0 100 0 ?
Over on PE3 through 2.2.2.3/32is now missing from the ADMIN VRF and exists inside the CUST VRF.
PE3#show bgp vpnv4 unicast all
BGP table version is 29, local router ID is 10.0.0.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:1
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
Route Distinguisher: 1.1.1.1:2
*>i 2.2.2.2/32 10.0.0.2 0 100 0 ?
*>i 2.2.2.3/32 10.0.0.2 0 100 0 ?
Route Distinguisher: 1.1.1.1:3 (default for vrf CUST)
*>i 2.2.2.1/32 10.0.0.1 0 100 0 ?
*> 2.2.2.2/32 0.0.0.0 0 32768 ?
* i 10.0.0.2 0 100 0 ?
*>i 2.2.2.3/32 10.0.0.2 0 100 0 ?
Route Distinguisher: 1.1.1.1:4 (default for vrf ADMIN)
*> 2.2.2.4/32 0.0.0.0 0 32768 ?
PE3#show bgp vpnv4 unicast all 2.2.2.3/32
BGP routing table entry for 1.1.1.1:2:2.2.2.3/32, version 15
Paths: (1 available, best #1, no table)
Not advertised to any peer
Refresh Epoch 1
Local
10.0.0.2 (metric 2) from 10.0.0.4 (10.0.0.4)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
Originator: 10.0.0.2, Cluster list: 10.0.0.4
mpls labels in/out nolabel/17
BGP routing table entry for 1.1.1.1:3:2.2.2.3/32, version 16
Paths: (1 available, best #1, table CUST)
Not advertised to any peer
Refresh Epoch 1
Local, imported path from 1.1.1.1:2:2.2.2.3/32
10.0.0.2 (metric 2) from 10.0.0.4 (10.0.0.4)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
Originator: 10.0.0.2, Cluster list: 10.0.0.4
mpls labels in/out nolabel/17
Because RR1 prefers 2.2.2.3/32 via PE2 (as shown below) that is what is advertised to PE3, and the PE2 version of the route has an RT of 1:1 which means PE3 imports the route into the CUST VRF and now, despite a new prefix being advertised by PE2 in the CUST L3 VPN, connectivity is lost between two subnets in a different L3 VPN (ADMIN) between two different devices (PE3 and ASBR1):
RR1#show bgp vpnv4 unicast all 2.2.2.3/32
BGP routing table entry for 1.1.1.1:2:2.2.2.3/32, version 9
Paths: (2 available, best #1, no table)
Advertised to update-groups:
1
Refresh Epoch 1
Local, (Received from a RR-client)
10.0.0.2 (metric 3) from 10.0.0.2 (10.0.0.2)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/17
Refresh Epoch 1
Local, (Received from a RR-client)
10.0.0.5 (metric 4) from 10.0.0.5 (10.0.0.5)
Origin incomplete, metric 0, localpref 100, valid, internal
Extended Community: RT:1:2
mpls labels in/out nolabel/16
The route-reflector problem can be exacerbated if Inter-AS MPLS Opt C is introduced, making peerings between inter-AS route-reflectors, where a typical Inter-AS option is to also disable route-target filtering on the ASBRs (“no bgp default route-target filter” on IOS or “retain route-target all” on IOS-XR). In actual fact this alone wouldn’t have an effect as the BGP best path selection process will still prefer the iBGP routes over eBGP routes from an Inter-AS connection. However, it can be common with mergers and acquisitions for example that a service should now be preferred via the “new” AS to migrate from a legacy platform or connection. A badly configured policy to improve the local preference for all prefixes learnt over the eBGP peering means any overlapping routes will be preferred over local AS routes and again, connectivity within L3 VPNs will seem to mysterious stop.
These examples show that the RD of a VRF is not completely locally significant. However that wording is part of the problem. The name of a VRF is truly “locally significant”. The RD is more or less locally significant, considering the example of two PEs originating the same prefix with the same RD, one of the paths from one PE was not visible to other PEs, so this has only affected routing to prefixes originated from that PE; thus it can be said that the RD is locally significant to that PE only with regards to reachability of prefixes originated from that PE. Perhaps it would be more accurate to say it has a global impact on reachability on locally originated prefixes.
It was shown that RDs and their uniqueness (or lack thereof) affects memory usage, the number of available paths (which can impact multi-pathing and backup paths availability), and the BGP best path calculation (on an RR and on a PE). The inter-VPN issues that were shown could be classed as “configuration mistakes” like configuring two overlapping prefixes on two separate interfaces on the same device, and generally RDs would be unique per PE and per VRF. There is no single right answer with RD allocation policies.
Previous page: Per Flow Load-Balancing Without FAT
Next page: VASI Inter-VPN Routing