Date created: Monday, February 26, 2018 4:13:22 PM. Last modified: Sunday, December 3, 2023 4:34:22 PM

EVPN with MPLS

References:

https://www.rfc-editor.org/rfc/rfc7432.html - BGP MPLS-Based Ethernet VPN
https://www.rfc-editor.org/rfc/rfc9012.html - The BGP Tunnel Encapsulation Attribute
https://www.cisco.com/c/en/us/td/docs/routers/ios/config/17-x/mpls/b-mpls/m-ce-evpn-multihoming.html#concept_wvp_rqh_l2b - Cisco IOS XE 17.x -> EVPN Multihoming

 

See Also

EVPN with VXLAN and NVGRE
VXLAN Recap

 

EVPN Forwarding Plane

EVPN can use VXLAN (Virtual eXtensible Local Area Network), NVGRE (Network Virtualization using Generic Routing Encapsulation), MPLS (Multi-Protocol Label Switching), or MPLS over GRE for data plane encapsulation.

NVGRE has the term Network Virtualization Edges (NVEs) which is equivalent to an MPLS PE (the VPN encap/decap edge device).

VXLAN has the term Tunnel End Points (VTEPs), which again is equivalent to an MPLS PE (edge encap/decap).

MPLS uses the term Provider Edge (PE) router or Label Edge Router.

VXLAN and NVGRE include an NVO ID (Network Virtualisation Overlay identifier) in the transport packet header, which identifies the specific overlay instance the inner packet relates to. This is called a VNI in VXLAN (VXLAN Network Identifier a.k.a VXLAN Segment ID) and a VSID in NVGRE (Virtual Subnet Identifier). For MPLS unique labels are allocated per overlay because there are no additional header fields to use.

 

EVPN Control Plane

To avoid the inefficient method of flood-and-learn for ARP and unknown unicast packets, control-plane signaling is used to to advertise layer 2 and layer 3 “routes”. EVPN can advertise: MAC addresses, MAC and IP bindings, and pure IP Prefixes:

  • EVPN only uses MP-BGP for control plane signaling.
  • BGP can be used to auto-discover other PEs/VTEPs.

In the VXLAN world, a VNI is a VXLAN Network Identifier in the VXLAN header which explicitly separates traffic sent between two VTEPs.

In the MPLS world, an EVI is an EVPN Instance which spans one or more PEs. A MAC-VRF is like a L3 VRF for IP VPNs. The Route Distinguisher configured on the MAC-VRF implicitly identifies the EVI the routers belong to when they are being advertised via BGP.

The following applies to both VXLAN and MPLs:

   The EVPN NLRI is carried in BGP [RFC4271] using BGP Multiprotocol
   Extensions [RFC4760] with an Address Family Identifier (AFI) of 25
   (L2VPN) and a Subsequent Address Family Identifier (SAFI) of 70
   (EVPN).  The NLRI field in the MP_REACH_NLRI/MP_UNREACH_NLRI
   attribute contains the EVPN NLRI (encoded as specified above).

   In order for two BGP speakers to exchange labeled EVPN NLRI, they
   must use BGP Capabilities Advertisements to ensure that they both are
   capable of properly processing such NLRI.  This is done as specified
   in [RFC4760], by using capability code 1 (multiprotocol BGP) with an
   AFI of 25 (L2VPN) and a SAFI of 70 (EVPN).

For VXLAN: The Next Hop field of the MP_REACH_NLRI attribute of the route MUST be set to the IPv4 or IPv6 address of the NVE.

For MPLS: The Next Hop field of the MP_REACH_NLRI attribute of the route MUST be set to the IPv4 or IPv6 address of the advertising PE.

The BGP Encapsulation Extended Community is used to signal the transport type (RFC9012). These are recorded by IANA: https://www.iana.org/assignments/bgp-tunnel-encapsulation/bgp-tunnel-encapsulation.xhtml#tunnel-types

8 - VXLAN
9 - NVGRE
10 - MPLS
11 - MPLS in GRE Encapsulation

 

EVPN with MPLS

The RFC7432 Approach "BGP MPLS-Based Ethernet VPN" - BGP Signaled EVPN with MPLS Transport

   EVI: An EVPN instance spanning the Provider Edge (PE) devices
      participating in that EVPN.

   MAC-VRF: A Virtual Routing and Forwarding table for Media Access
      Control (MAC) addresses on a PE. This is identified by a Route Distinguisher and uses one or more Route Targets for importing and exporting EVPN routes.

   ES:  Ethernet Segment.  When a customer site (device or network) is
      connected to one or more PEs via a set of Ethernet links, then
      that set of links is referred to as an 'Ethernet segment'.

   Ethernet Segment Identifier (ESI):  A unique non-zero identifier that
      identifies an Ethernet segment is called an 'Ethernet Segment
      Identifier'. An ESI is encoded as a 10-octet integer in
   line format with the most significant octet sent first.  The
   following two ESI values are reserved:

   - ESI 0 denotes a single-homed site.

   - ESI {0xFF} (repeated 10 times) is known as MAX-ESI and is reserved.

The ESI has the  following format:

               +---+---+---+---+---+---+---+---+---+---+
               | T |          ESI Value                |
               +---+---+---+---+---+---+---+---+---+---+

   Where:

   T (ESI Type) is a 1-octet field (most significant octet) that
   specifies the format of the remaining 9 octets (ESI Value).  The
   following six ESI types can be used:

   - Type 0 (T=0x00) - This type indicates an arbitrary 9-octet ESI
     value, which is managed and configured by the operator.

   - Type 1 (T=0x01) - When IEEE 802.1AX LACP is used between the PEs
     and CEs, this ESI type indicates an auto-generated ESI value
     determined from LACP.

   - Type 2 (T=0x02) - This type is used in the case of indirectly
     connected hosts via a bridged LAN between the CEs and the PEs.  The
     ESI Value is auto-generated and determined based on the Layer 2
     bridge protocol.

   - Type 3 (T=0x03) - This type indicates a MAC-based ESI Value that
     can be auto-generated or configured by the operator. 

   - Type 4 (T=0x04) - This type indicates a router-ID ESI Value that
     can be auto-generated or configured by the operator.

   - Type 5 (T=0x05) - This type indicates an Autonomous System
     (AS)-based ESI Value that can be auto-generated or configured by
     the operator.


6.  Ethernet Tag ID

   An Ethernet Tag ID is a 32-bit field containing either a 12-bit or
   24-bit identifier that identifies a particular broadcast domain
   (e.g., a VLAN) in an EVPN instance.  The 12-bit identifier is called
   the VLAN ID (VID).  An EVPN instance consists of one or more
   broadcast domains (one or more VLANs).  VLANs are assigned to a given
   EVPN instance by the provider of the EVPN service.  A given VLAN can
   itself be represented by multiple VIDs.  In such cases, the PEs
   participating in that VLAN for a given EVPN instance are responsible
   for performing VLAN ID translation to/from locally attached CE
   devices.

   If a VLAN is represented by a single VID across all PE devices
   participating in that VLAN for that EVPN instance, then there is no
   need for VID translation at the PEs. Furthermore, some deployment
   scenarios guarantee uniqueness of VIDs across all EVPN instances; all
   points of attachment for a given EVPN instance use the same VID, and
   no other EVPN instances use that VID.  This allows the RT(s) for each
   EVPN instance to be derived automatically from the corresponding VID,
   as described in Section 7.10.1.

   The following subsections discuss the relationship between broadcast
   domains (e.g., VLANs), Ethernet Tag IDs (e.g., VIDs), and MAC-VRFs as
   well as the setting of the Ethernet Tag ID, in the various EVPN BGP
   routes (defined in Section 8), for the different types of service
   interfaces described in [RFC7209].
   The following Ethernet Tag ID value is reserved:

   - Ethernet Tag ID {0xFFFFFFFF} is known as MAX-ET.

6.1.  VLAN-Based Service Interface

   With this service interface, an EVPN instance consists of only a
   single broadcast domain (e.g., a single VLAN).  Therefore, there is a
   one-to-one mapping between a VID on this interface and a MAC-VRF.
   Since a MAC-VRF corresponds to a single VLAN, it consists of a single
   bridge table corresponding to that VLAN.  If the VLAN is represented
   by multiple VIDs (e.g., a different VID per Ethernet segment per PE),
   then each PE needs to perform VID translation for frames destined to
   its Ethernet segment(s).  In such scenarios, the Ethernet frames
   transported over an MPLS/IP network SHOULD remain tagged with the
   originating VID, and a VID translation MUST be supported in the data
   path and MUST be performed on the disposition PE.  The Ethernet Tag
   ID in all EVPN routes MUST be set to 0.

 

7.  BGP EVPN Routes

0 – This is reserved and not used (RFC7432)
1 - Ethernet Auto-Discovery (A-D) route (RFC7432)
2 - MAC/IP Advertisement route (RFC7432)
3 - Inclusive Multicast Ethernet Tag route (RFC7432)
4 - Ethernet Segment route (RFC7432)
5 – IP Prefix Route (RFC9136)
…
   In order for two BGP speakers to exchange labeled EVPN NLRI, they
   must use BGP Capabilities Advertisements to ensure that they both are
   capable of properly processing such NLRI.  This is done as specified
   in [RFC4760], by using capability code 1 (multiprotocol BGP) with an
   AFI of 25 (L2VPN) and a SAFI of 70 (EVPN).

From FRR:

EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

There are two sub-types of Type 1 Ethernet Auto Discovery routes. They are not defined in the RFC. The RFC states that the only the ESI and Tag ID form the "route" within the NLRI, the MPLS label is not included:

7.1.  Ethernet Auto-discovery Route

   An Ethernet A-D route type specific EVPN NLRI consists of the
   following:

                +---------------------------------------+
                |  Route Distinguisher (RD) (8 octets)  |
                +---------------------------------------+
                |Ethernet Segment Identifier (10 octets)|
                +---------------------------------------+
                |  Ethernet Tag ID (4 octets)           |
                +---------------------------------------+
                |  MPLS Label (3 octets)                |
                +---------------------------------------+

   For the purpose of BGP route key processing, only the Ethernet
   Segment Identifier and the Ethernet Tag ID are considered to be part
   of the prefix in the NLRI.  The MPLS Label field is to be treated as
   a route attribute as opposed to being part of the route.

 

From: https://www.cisco.com/c/en/us/td/docs/routers/ios/config/17-x/mpls/b-mpls/m-ce-evpn-multihoming.html#concept_wvp_rqh_l2b

The MPLS label is set to 0 to indicate a Per-ES Ethernet Auto-Discovery Route. Any non-zero value indicates a Per-EVI Ethernet Auto-Discovery Route.

This route is originated when a PE is connected to a CE for which multihoming is configured. Route Type 1 is advertised only if ES is set to a non-zero value, that is, type 1 routes are originated only for sites where multihoming is configured. These routes are sent per-ES and carry the combined set of route targets of all of the EVIs that belong to that ES.

Two types of EAD routes are supported in this feature: Per-EVI (EVPN Instance) EAD routes and Per-ES (Ethernet Segment) EAD Routes. Route Type-1 advertisement is used for achieving split-horizon, fast convergence, and aliasing. EAD-ES and EAD-EVI are used to achieve these functionalities. The per-ES EAD route includes the ESI label extended community which indicates if it is an all-active or a single-active configuration. The ESI label extended community also carries the ESI label that is used for split horizon configuration. The per-ES EAD route is also used for fast convergence when failure occurs at the ES on the access side. The per-EVI EAD and per-ES EAD routes are used for aliasing, and fast convergence and providing the split horizon label, respectively. In a multihoming group, each PE associated with a CE may learn only a subset of MAC addresses on traffic ingress from CE. The MAC addresses learned by the PEs may not overlap with each other. Aliasing is the ability of a PE to signal that it has reachability to an EVPN instance on a given ES, even when the PE has not learned MAC addresses from that EVI or ES. In an all-active multi-homing configuration, a remote PE that receives a MAC advertisement route considers the advertised MAC address to be reachable through all PEs that have advertised reachability to EVI or ES of the MAC address.

One per-ES-EAD route is sourced per Ethernet Segment. Per-ES-EAD route carries the route targets of all EVIs the Ethernet Segment belongs to. If the number of EVI route targets is too large to be carried in one per-ES-EAD route, then multiple routes are advertised. Each route is assigned a different Ethernet Segment Route Distinguisher (ES-RD). The per-EVI-EAD route is used along with the per-ES-EAD route for aliasing and backup path. The per-ES-EAD is also used for fast convergence in case of failure in the Ethernet Segment.

 

7.2.  MAC/IP Advertisement Route

   A MAC/IP Advertisement route type specific EVPN NLRI consists of the
   following:

                +---------------------------------------+
                |  RD (8 octets)                        |
                +---------------------------------------+
                |Ethernet Segment Identifier (10 octets)|
                +---------------------------------------+
                |  Ethernet Tag ID (4 octets)           |
                +---------------------------------------+
                |  MAC Address Length (1 octet)         |
                +---------------------------------------+
                |  MAC Address (6 octets)               |
                +---------------------------------------+
                |  IP Address Length (1 octet)          |
                +---------------------------------------+
                |  IP Address (0, 4, or 16 octets)      |
                +---------------------------------------+
                |  MPLS Label1 (3 octets)               |
                +---------------------------------------+
                |  MPLS Label2 (0 or 3 octets)          |
                +---------------------------------------+


   For the purpose of BGP route key processing, only the Ethernet Tag
   ID, MAC Address Length, MAC Address, IP Address Length, and IP
   Address fields are considered to be part of the prefix in the NLRI.
   The Ethernet Segment Identifier, MPLS Label1, and MPLS Label2 fields
   are to be treated as route attributes as opposed to being part of the
   "route".  Both the IP and MAC address lengths are in bits.

7.4.  Ethernet Segment Route

   An Ethernet Segment route type specific EVPN NLRI consists of the
   following:

               +---------------------------------------+
               |  RD (8 octets)                        |
               +---------------------------------------+
               |Ethernet Segment Identifier (10 octets)|
               +---------------------------------------+
               |  IP Address Length (1 octet)          |
               +---------------------------------------+
               |  Originating Router's IP Address      |
               |          (4 or 16 octets)             |
               +---------------------------------------+
   For the purpose of BGP route key processing, only the Ethernet
   Segment ID, IP Address Length, and Originating Router's IP Address
   fields are considered to be part of the prefix in the NLRI. The IP address length is in bits.

7.5.  ESI Label Extended Community

   This Extended Community is a new transitive Extended Community having
   a Type field value of 0x06 and the Sub-Type 0x01.  It may be
   advertised along with Ethernet Auto-discovery routes, and it enables
   split-horizon procedures for multihomed sites as described in
   Section 8.3 ("Split Horizon").  The ESI Label field represents an ES
   by the advertising PE, and it is used in split-horizon filtering by
   other PEs that are connected to the same multihomed Ethernet segment.

   Each ESI Label extended community is encoded as an 8-octet value, as
   follows:

     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Type=0x06     | Sub-Type=0x01 | Flags(1 octet)|  Reserved=0   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  Reserved=0   |          ESI Label                            |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   The low-order bit of the Flags octet is defined as the
   "Single-Active" bit.  A value of 0 means that the multihomed site
   is operating in All-Active redundancy mode, and a value of 1 means
   that the multihomed site is operating in Single-Active redundancy
   mode.

7.6.  ES-Import Route Target

   This is a new transitive Route Target extended community carried with
   the Ethernet Segment route.  When used, it enables all the PEs
   connected to the same multihomed site to import the Ethernet Segment
   routes.  The value is derived automatically for the ESI Types 1, 2,
   and 3, by encoding the high-order 6-octet portion of the 9-octet ESI
   Value, which corresponds to a MAC address, in the ES-Import Route
   Target.  The format of this Extended Community is as follows:

     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Type=0x06     | Sub-Type=0x02 |          ES-Import            |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                     ES-Import Cont'd                          |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   This document expands the definition of the Route Target extended
   community to allow the value of the high-order octet (Type field) to
   be 0x06 (in addition to the values specified in [RFC4360]).  The
   low-order octet (Sub-Type field) value 0x02 indicates that this
   Extended Community is of type "Route Target".  The new Type field
   value 0x06 indicates that the structure of this RT is a 6-octet value
   (e.g., a MAC address).  A BGP speaker that implements RT Constraint
   [RFC4684] MUST apply the RT Constraint procedures to the ES-Import RT
   as well.

8.2.  Fast Convergence
   ...EVPN defines a mechanism to efficiently and
   quickly signal, to remote PE nodes, the need to update their
   forwarding tables upon the occurrence of a failure in connectivity to
   an Ethernet segment.  This is done by having each PE advertise a set
   of one or more Ethernet A-D per ES routes for each locally attached
   Ethernet segment (refer to Section 8.2.1 below for details on how
   these routes are constructed).  A PE may need to advertise more than
   one Ethernet A-D per ES route for a given ES because the ES may be in
   a multiplicity of EVIs and the RTs for all of these EVIs may not fit
   into a single route.  Advertising a set of Ethernet A-D per ES routes
   for the ES allows each route to contain a subset of the complete set
   of RTs.  Each Ethernet A-D per ES route is differentiated from the
   other routes in the set by a different Route Distinguisher (RD).

   Upon a failure in connectivity to the attached segment, the PE
   withdraws the corresponding set of Ethernet A-D per ES routes.  This
   triggers all PEs that receive the withdrawal to update their next-hop
   adjacencies for all MAC addresses associated with the Ethernet
   segment in question.  If no other PE had advertised an Ethernet A-D
   route for the same segment, then the PE that received the withdrawal
   simply invalidates the MAC entries for that segment.  Otherwise, the
   PE updates its next-hop adjacencies accordingly.

8.2.1.  Constructing Ethernet A-D per Ethernet Segment Route

   This section describes the procedures used to construct the Ethernet
   A-D per ES route, which is used for fast convergence (as discussed
   above) and for advertising the ESI label used for split-horizon
   filtering (as discussed in Section 8.3).  Support of this route is
   REQUIRED.

   The Route Distinguisher (RD) MUST be a Type 1 RD [RFC4364].  The
   value field comprises an IP address of the PE (typically, the
   loopback address) followed by a number unique to the PE.

   The Ethernet Segment Identifier MUST be a 10-octet entity as
   described in Section 5 ("Ethernet Segment").  The Ethernet A-D route
   is not needed when the Segment Identifier is set to 0 (e.g., single-
   homed scenarios).

   The Ethernet Tag ID MUST be set to MAX-ET.

   The MPLS label in the NLRI MUST be set to 0.

   The ESI Label extended community MUST be included in the route.  If
   All-Active redundancy mode is desired, then the "Single-Active" bit
   in the flags of the ESI Label extended community MUST be set to 0 and
   the MPLS label in that Extended Community MUST be set to a valid MPLS
   label value.  The MPLS label in this Extended Community is referred
   to as the ESI label and MUST have the same value in each Ethernet A-D
   per ES route advertised for the ES.  This label MUST be a downstream
   assigned MPLS label if the advertising PE is using ingress
   replication for receiving multicast, broadcast, or unknown unicast
   traffic from other PEs.  If the advertising PE is using P2MP MPLS
   LSPs for sending multicast, broadcast, or unknown unicast traffic,
   then this label MUST be an upstream assigned MPLS label.  The usage
   of this label is described in Section 8.3.

   If Single-Active redundancy mode is desired, then the "Single-Active"
   bit in the flags of the ESI Label extended community MUST be set to 1
   and the ESI label SHOULD be set to a valid MPLS label value.

8.3.  Split Horizon

   Consider a CE that is multihomed to two or more PEs on an Ethernet
   segment ES1 operating in All-Active redundancy mode.  If the CE sends
   a broadcast, unknown unicast, or multicast (BUM) packet to one of the
   non-Designated Forwarder (non-DF) PEs, say PE1, then PE1 will forward
   that packet to all or a subset of the other PEs in that EVPN
   instance, including the DF PE for that Ethernet segment.  In this
   case, the DF PE to which the CE is multihomed MUST drop the packet
   and not forward back to the CE.  This filtering is referred to as
   "split-horizon filtering" in this document.

   When a set of PEs are operating in Single-Active redundancy mode, the
   use of this split-horizon filtering mechanism is highly recommended
   because it prevents transient loops at the time of failure or
   recovery that would impact the Ethernet segment -- e.g., when two PEs
   think that both are DFs for that segment before the DF election
   procedure settles down.

   In order to achieve this split-horizon function, every BUM packet
   originating from a non-DF PE is encapsulated with an MPLS label that
   identifies the Ethernet segment of origin (i.e., the segment from
   which the frame entered the EVPN network).  This label is referred to
   as the ESI label and MUST be distributed by all PEs when operating in
   All-Active redundancy mode using a set of Ethernet A-D per ES routes,
   per Section 8.2.1 above.  The ESI label SHOULD be distributed by all
   PEs when operating in Single-Active redundancy mode using a set of
   Ethernet A-D per ES routes.  These routes are imported by the PEs
   connected to the Ethernet segment and also by the PEs that have at
   least one EVPN instance in common with the Ethernet segment in the
   route.  As described in Section 8.1.1, the route MUST carry an ESI
   Label extended community with a valid ESI label.  The disposition PE
   relies on the value of the ESI label to determine whether or not a
   BUM frame is allowed to egress a specific Ethernet segment.

8.3.1.  ESI Label Assignment

   The following subsections describe the assignment procedures for the
   ESI label, which differ depending on the type of tunnels being used
   to deliver multi-destination packets in the EVPN network.

8.3.1.1.  Ingress Replication

   Each PE that operates in All-Active or Single-Active redundancy mode
   and that uses ingress replication to receive BUM traffic advertises a
   downstream assigned ESI label in the set of Ethernet A-D per ES
   routes for its attached ES.  This label MUST be programmed in the
   platform label space by the advertising PE, and the forwarding entry
   for this label must result in NOT forwarding packets received with
   this label onto the Ethernet segment for which the label was
   distributed.

   The rules for the inclusion of the ESI label in a BUM packet by the
   ingress PE operating in All-Active redundancy mode are as follows:

   - A non-DF ingress PE MUST include the ESI label distributed by the
     DF egress PE in the copy of a BUM packet sent to it.

   - An ingress PE (DF or non-DF) SHOULD include the ESI label
     distributed by each non-DF egress PE in the copy of a BUM packet
     sent to it.

   The rule for the inclusion of the ESI label in a BUM packet by the
   ingress PE operating in Single-Active redundancy mode is as follows:

   - An ingress DF PE SHOULD include the ESI label distributed by the
     egress PE in the copy of a BUM packet sent to it.

   In both All-Active and Single-Active redundancy mode, an ingress PE
   MUST NOT include an ESI label in the copy of a BUM packet sent to an
   egress PE that is not attached to the ES through which the BUM packet
   entered the EVI.

   As an example, consider PE1 and PE2, which are multihomed to CE1 on
   ES1 and operating in All-Active multihoming mode.  Further, consider
   that PE1 is using P2P or MP2P LSPs to send packets to PE2.  Consider
   that PE1 is the non-DF for VLAN1 and PE2 is the DF for VLAN1, and PE1
   receives a BUM packet from CE1 on VLAN1 on ES1.  In this scenario,
   PE2 distributes an Inclusive Multicast Ethernet Tag route for VLAN1
   corresponding to an EVPN instance.  So, when PE1 sends a BUM packet
   that it receives from CE1, it MUST first push onto the MPLS label
   stack the ESI label that PE2 has distributed for ES1.  It MUST then
   push onto the MPLS label stack the MPLS label distributed by PE2 in
   the Inclusive Multicast Ethernet Tag route for VLAN1.  The resulting
   packet is further encapsulated in the P2P or MP2P LSP label stack
   required to transmit the packet to PE2.  When PE2 receives this
   packet, it determines, from the top MPLS label, the set of ESIs to
   which it will replicate the packet after any P2P or MP2P LSP labels
   have been removed.  If the next label is the ESI label assigned by
   PE2 for ES1, then PE2 MUST NOT forward the packet onto ES1.  If the
   next label is an ESI label that has not been assigned by PE2, then
   PE2 MUST drop the packet.  It should be noted that in this scenario,
   if PE2 receives a BUM packet for VLAN1 from CE1, then it SHOULD
   encapsulate the packet with an ESI label received from PE1 when
   sending it to PE1 in order to avoid any transient loops during a
   failure scenario that would impact ES1 (e.g., port or link failure).

8.3.1.2.  P2MP MPLS LSPs

   The non-DF PEs that operate in All-Active redundancy mode and that
   use P2MP LSPs to send BUM traffic advertise an upstream assigned ESI
   label in the set of Ethernet A-D per ES routes for their common
   attached ES.  This label is upstream assigned by the PE that
   advertises the route.  This label MUST be programmed by the other PEs
   that are connected to the ESI advertised in the route, in the context
   label space for the advertising PE.  Further, the forwarding entry
   for this label must result in NOT forwarding packets received with
   this label onto the Ethernet segment for which the label was
   distributed.  This label MUST also be programmed by the other PEs
   that import the route but are not connected to the ESI advertised in
   the route, in the context label space for the advertising PE.
   Further, the forwarding entry for this label must be a label pop with
   no other associated action.

   The DF PE that operates in Single-Active redundancy mode and that
   uses P2MP LSPs to send BUM traffic should advertise an upstream
   assigned ESI label in the set of Ethernet A-D per ES routes for its
   attached ES, just as described in the previous paragraph.

   As an example, consider PE1 and PE2, which are multihomed to CE1 on
   ES1 and operating in All-Active multihoming mode.  Also, consider
   that PE3 belongs to one of the EVPN instances of ES1.  Further,
   assume that PE1, which is the non-DF, is using P2MP MPLS LSPs to send
   BUM packets.  When PE1 sends a BUM packet that it receives from CE1,
   it MUST first push onto the MPLS label stack the ESI label that it
   has assigned for the ESI on which the packet was received.  The
   resulting packet is further encapsulated in the P2MP MPLS label stack
   necessary to transmit the packet to the other PEs.  Penultimate hop
   popping MUST be disabled on the P2MP LSPs used in the MPLS transport
   infrastructure for EVPN.  When PE2 receives this packet, it
   decapsulates the top MPLS label and forwards the packet using the
   context label space determined by the top label.  If the next label
   is the ESI label assigned by PE1 to ES1, then PE2 MUST NOT forward
   the packet onto ES1.  When PE3 receives this packet, it decapsulates
   the top MPLS label and forwards the packet using the context label
   space determined by the top label.  If the next label is the ESI
   label assigned by PE1 to ES1 and PE3 is not connected to ES1, then
   PE3 MUST pop the label and flood the packet over all local ESIs in
   that EVPN instance.  It should be noted that when PE2 sends a BUM
   frame over a P2MP LSP, it should encapsulate the frame with an ESI
   label even though it is the DF for that VLAN, in order to avoid any
   transient loops during a failure scenario that would impact ES1
   (e.g., port or link failure).

8.4.  Aliasing and Backup Path

   In the case where a CE is multihomed to multiple PE nodes, using a
   Link Aggregation Group (LAG) with All-Active redundancy, it is
   possible that only a single PE learns a set of the MAC addresses
   associated with traffic transmitted by the CE.  This leads to a
   situation where remote PE nodes receive MAC/IP Advertisement routes
   for these addresses from a single PE, even though multiple PEs are
   connected to the multihomed segment.  As a result, the remote PEs are
   not able to effectively load balance traffic among the PE nodes
   connected to the multihomed Ethernet segment.  This could be the
   case, for example, when the PEs perform data-plane learning on the
   access, and the load-balancing function on the CE hashes traffic from
   a given source MAC address to a single PE.

   Another scenario where this occurs is when the PEs rely on control-
   plane learning on the access (e.g., using ARP), since ARP traffic
   will be hashed to a single link in the LAG.

   To address this issue, EVPN introduces the concept of 'aliasing',
   which is the ability of a PE to signal that it has reachability to an
   EVPN instance on a given ES even when it has learned no MAC addresses
   from that EVI/ES.  The Ethernet A-D per EVI route is used for this
   purpose.  A remote PE that receives a MAC/IP Advertisement route with
   a non-reserved ESI SHOULD consider the advertised MAC address to be
   reachable via all PEs that have advertised reachability to that MAC
   address's EVI/ES via the combination of an Ethernet A-D per EVI route
   for that EVI/ES (and Ethernet tag, if applicable) AND Ethernet A-D
   per ES routes for that ES with the "Single-Active" bit in the flags
   of the ESI Label extended community set to 0.

   Note that the Ethernet A-D per EVI route may be received by a remote
   PE before it receives the set of Ethernet A-D per ES routes.
   Therefore, in order to handle corner cases and race conditions, the
   Ethernet A-D per EVI route MUST NOT be used for traffic forwarding by
   a remote PE until it also receives the associated set of Ethernet A-D
   per ES routes.

   The backup path is a closely related function, but it is used in
   Single-Active redundancy mode.  In this case, a PE also advertises
   that it has reachability to a given EVI/ES using the same combination
   of Ethernet A-D per EVI route and Ethernet A-D per ES route as
   discussed above, but with the "Single-Active" bit in the flags of the
   ESI Label extended community set to 1.  A remote PE that receives a
   MAC/IP Advertisement route with a non-reserved ESI SHOULD consider
   the advertised MAC address to be reachable via any PE that has
   advertised this combination of Ethernet A-D routes, and it SHOULD
   install a backup path for that MAC address.

8.4.1.  Constructing Ethernet A-D per EVPN Instance Route

   This section describes the procedures used to construct the Ethernet
   A-D per EVPN instance (EVI) route, which is used for aliasing (as
   discussed above).  Support of this route is OPTIONAL.

   The Route Distinguisher (RD) MUST be set per Section 7.9.

   The Ethernet Segment Identifier MUST be a 10-octet entity as
   described in Section 5 ("Ethernet Segment").  The Ethernet A-D route
   is not needed when the Segment Identifier is set to 0.

   The Ethernet Tag ID is the identifier of an Ethernet tag on the
   Ethernet segment.  This value may be a 12-bit VLAN ID, in which case
   the low-order 12 bits are set to the VLAN ID and the high-order
   20 bits are set to 0.  Or, it may be another Ethernet tag used by the
   EVPN.  It MAY be set to the default Ethernet tag on the Ethernet
   segment or to the value 0.

   Note that the above allows the Ethernet A-D route to be advertised
   with one of the following granularities:

   + One Ethernet A-D route per <ESI, Ethernet Tag ID> tuple per
     MAC-VRF.  This is applicable when the PE uses MPLS-based
     disposition with VID translation or may be applicable when the
     PE uses MAC-based disposition with VID translation.

   + One Ethernet A-D route for each  per MAC-VRF (where the
     Ethernet Tag ID is set to 0).  This is applicable when the PE uses
     MAC-based disposition or MPLS-based disposition without VID
     translation.

   The usage of the MPLS label is described in Section 14 ("Load
   Balancing of Unicast Packets").

   The Next Hop field of the MP_REACH_NLRI attribute of the route MUST
   be set to the IPv4 or IPv6 address of the advertising PE.

   The Ethernet A-D route MUST carry one or more Route Target (RT)
   attributes, per Section 7.10.

8.5.  Designated Forwarder Election

   Consider a CE that is a host or a router that is multihomed directly
   to more than one PE in an EVPN instance on a given Ethernet segment.
   One or more Ethernet tags may be configured on the Ethernet segment.
   In this scenario, only one of the PEs, referred to as the Designated
   Forwarder (DF), is responsible for certain actions:

   - Sending multicast and broadcast traffic, on a given Ethernet tag on
     a particular Ethernet segment, to the CE.

   - Flooding unknown unicast traffic (i.e., traffic for which a PE does
     not know the destination MAC address), on a given Ethernet tag on a
     particular Ethernet segment to the CE, if the environment requires
     flooding of unknown unicast traffic.

   Note that this behavior, which allows selecting a DF at the
   granularity of <ES, VLAN> or <ES, VLAN bundle> for multicast,
   broadcast, and unknown unicast traffic, is the default behavior in
   this specification.

   Note that a CE always sends packets belonging to a specific flow
   using a single link towards a PE.  For instance, if the CE is a host,
   then, as mentioned earlier, the host treats the multiple links that
   it uses to reach the PEs as a Link Aggregation Group (LAG).  The CE
   employs a local hashing function to map traffic flows onto links in
   the LAG.

   If a bridged network is multihomed to more than one PE in an EVPN
   network via switches, then the support of All-Active redundancy mode
   requires the bridged network to be connected to two or more PEs using
   a LAG.

   If a bridged network does not connect to the PEs using a LAG, then
   only one of the links between the bridged network and the PEs must be
   the active link for a given <ES, VLAN> or <ES, VLAN bundle>.  In this
   case, the set of Ethernet A-D per ES routes advertised by each PE
   MUST have the "Single-Active" bit in the flags of the ESI Label
   extended community set to 1.

   The default procedure for DF election at the granularity of <ES,
   VLAN> for VLAN-based service or <ES, VLAN bundle> for VLAN-(aware)
   bundle service is referred to as "service carving".  With service
   carving, it is possible to elect multiple DFs per Ethernet segment
   (one per VLAN or VLAN bundle) in order to perform load balancing of
   multi-destination traffic destined to a given segment.  The load-
   balancing procedures carve up the VLAN space per ES among the PE

   nodes evenly, in such a way that every PE is the DF for a disjoint
   set of VLANs or VLAN bundles for that ES.  The procedure for service
   carving is as follows:

   1. When a PE discovers the ESI of the attached Ethernet segment, it
      advertises an Ethernet Segment route with the associated ES-Import
      extended community attribute.

   2. The PE then starts a timer (default value = 3 seconds) to allow
      the reception of Ethernet Segment routes from other PE nodes
      connected to the same Ethernet segment.  This timer value should
      be the same across all PEs connected to the same Ethernet segment.

   3. When the timer expires, each PE builds an ordered list of the IP
      addresses of all the PE nodes connected to the Ethernet segment
      (including itself), in increasing numeric value.  Each IP address
      in this list is extracted from the "Originating Router's IP
      address" field of the advertised Ethernet Segment route.  Every PE
      is then given an ordinal indicating its position in the ordered
      list, starting with 0 as the ordinal for the PE with the
      numerically lowest IP address.  The ordinals are used to determine
      which PE node will be the DF for a given EVPN instance on the
      Ethernet segment, using the following rule:

      Assuming a redundancy group of N PE nodes, for VLAN-based service,
      the PE with ordinal i is the DF for an <ES, VLAN V> when (V mod N)
      = i.  In the case of VLAN-(aware) bundle service, then the
      numerically lowest VLAN value in that bundle on that ES MUST be
      used in the modulo function.

      It should be noted that using the "Originating Router's IP
      address" field in the Ethernet Segment route to get the PE IP
      address needed for the ordered list allows for a CE to be
      multihomed across different ASes if such a need ever arises.

   4. The PE that is elected as a DF for a given <ES, VLAN> or <ES, VLAN
      bundle> will unblock multi-destination traffic for that VLAN or
      VLAN bundle on the corresponding ES.  Note that the DF PE unblocks
      multi-destination traffic in the egress direction towards the
      segment.  All non-DF PEs continue to drop multi-destination
      traffic in the egress direction towards that <ES, VLAN> or <ES,
      VLAN bundle>.

      In the case of link or port failure, the affected PE withdraws its
      Ethernet Segment route.  This will re-trigger the service carving
      procedures on all the PEs in the redundancy group.  For PE node
      failure, or upon PE commissioning or decommissioning, the PEs
      re-trigger the service carving.  In the case of Single-Active
      multihoming, when a service moves from one PE in the redundancy
      group to another PE as a result of re-carving, the PE, which ends
      up being the elected DF for the service, SHOULD trigger a MAC
      address flush notification towards the associated Ethernet
      segment.  This can be done, for example, using the IEEE 802.1ak
      Multiple VLAN Registration Protocol (MVRP) 'new' declaration.

8.6.  Interoperability with Single-Homing PEs

   Let's refer to PEs that only support single-homed CE devices as
   single-homing PEs.  For single-homing PEs, all the above multihoming
   procedures can be omitted; however, to allow for single-homing PEs
   to fully interoperate with multihoming PEs, some of the multihoming
   procedures described above SHOULD be supported even by single-
   homing PEs.


9.2.1.  Constructing MAC/IP Address Advertisement

   BGP is extended to advertise these MAC addresses using the MAC/IP
   Advertisement route type in the EVPN NLRI.

   The RD MUST be set per Section 7.9.

   The Ethernet Segment Identifier is set to the 10-octet ESI described
   in Section 5 ("Ethernet Segment").

   The Ethernet Tag ID may be zero or may represent a valid Ethernet
   Tag ID.  This field may be non-zero when there are multiple bridge
   tables in the MAC-VRF (i.e., the PE needs to support VLAN-aware
   bundle service for that EVI).

   When the Ethernet Tag ID in the NLRI is set to a non-zero value for a
   particular broadcast domain, then this Ethernet Tag ID may be either
   the CE's Ethernet tag value (e.g., CE VLAN ID) or the EVPN provider's
   Ethernet tag value (e.g., provider VLAN ID).  The latter would be the
   case if the CE Ethernet tags (e.g., CE VLAN ID) for a particular
   broadcast domain are different on different CEs.

   The MAC Address Length field is in bits, and it is set to 48.  MAC
   address length values other than 48 bits are outside the scope of
   this document.  The encoding of a MAC address MUST be the 6-octet MAC
   address specified by [802.1Q] and [802.1D-REV].

   The IP Address field is optional.  By default, the IP Address Length
   field is set to 0, and the IP Address field is omitted from the
   route.  When a valid IP address needs to be advertised, it is then
   encoded in this route.  When an IP address is present, the IP Address
   Length field is in bits, and it is set to 32 or 128 bits.  Other IP
   Address Length values are outside the scope of this document.  The
   encoding of an IP address MUST be either 4 octets for IPv4 or
   16 octets for IPv6.  The Length field of the EVPN NLRI (which is in
   octets and is described in Section 7) is sufficient to determine
   whether an IP address is encoded in this route and, if so, whether
   the encoded IP address is IPv4 or IPv6.

   The MPLS Label1 field is encoded as 3 octets, where the high-order
   20 bits contain the label value.  The MPLS Label1 MUST be downstream
   assigned, and it is associated with the MAC address being advertised
   by the advertising PE.  The advertising PE uses this label when it
   receives an MPLS-encapsulated packet to perform forwarding based on
   the destination MAC address toward the CE.  The forwarding procedures
   are specified in Sections 13 and 14.

   A PE may advertise the same single EVPN label for all MAC addresses
   in a given MAC-VRF.  This label assignment is referred to as a per
   MAC-VRF label assignment.  Alternatively, a PE may advertise a unique
   EVPN label per <MAC-VRF, Ethernet tag> combination.  This label
   assignment is referred to as a per <MAC-VRF, Ethernet tag> label
   assignment.  As a third option, a PE may advertise a unique EVPN
   label per <ESI, Ethernet tag> combination.  This label assignment is
   referred to as a per <ESI, Ethernet tag> label assignment.  As a
   fourth option, a PE may advertise a unique EVPN label per MAC
   address.  This label assignment is referred to as a per MAC label
   assignment.  All of these label assignment methods have their
   trade-offs.  The choice of a particular label assignment methodology
   is purely local to the PE that originates the route.

   An assignment per MAC-VRF label requires the least number of EVPN
   labels but requires a MAC lookup in addition to an MPLS lookup on an
   egress PE for forwarding.  On the other hand, a unique label per
   <ESI, Ethernet tag> or a unique label per MAC allows an egress PE to
   forward a packet that it receives from another PE, to the connected
   CE, after looking up only the MPLS labels without having to perform a
   MAC lookup.  This includes the capability to perform appropriate VLAN
   ID translation on egress to the CE.

   The MPLS Label2 field is an optional field.  If it is present, then
   it is encoded as 3 octets, where the high-order 20 bits contain the
   label value.

   The Next Hop field of the MP_REACH_NLRI attribute of the route MUST
   be set to the IPv4 or IPv6 address of the advertising PE.

   The BGP advertisement for the MAC/IP Advertisement route MUST also
   carry one or more Route Target (RT) attributes.  RTs may be
   configured (as in IP VPNs) or may be derived automatically from the
   Ethernet Tag ID, in the Unique VLAN case, as described in
   Section 7.10.1.

   It is to be noted that this document does not require PEs to create
   forwarding state for remote MACs when they are learned in the control
   plane.  When this forwarding state is actually created is a local
   implementation matter.

9.2.2.  Route Resolution

   If the Ethernet Segment Identifier field in a received MAC/IP
   Advertisement route is set to the reserved ESI value of 0 or MAX-ESI,
   then if the receiving PE decides to install forwarding state for the
   associated MAC address, it MUST be based on the MAC/IP Advertisement
   route alone.

   If the Ethernet Segment Identifier field in a received MAC/IP
   Advertisement route is set to a non-reserved ESI, and the receiving
   PE is locally attached to the same ESI, then the PE does not alter
   its forwarding state based on the received route.  This ensures that
   local routes are preferred to remote routes.

   If the Ethernet Segment Identifier field in a received MAC/IP
   Advertisement route is set to a non-reserved ESI, then if the
   receiving PE decides to install forwarding state for the associated
   MAC address, it MUST be when both the MAC/IP Advertisement route AND
   the associated set of Ethernet A-D per ES routes have been received.
   The dependency of MAC route installation on Ethernet A-D per ES
   routes is to ensure that MAC routes don't get accidentally installed
   during a mass withdraw period.

10.  ARP and ND

   The IP Address field in the MAC/IP Advertisement route may optionally
   carry one of the IP addresses associated with the MAC address.  This
   provides an option that can be used to minimize the flooding of ARP

   or Neighbor Discovery (ND) messages over the MPLS network and to
   remote CEs.  This option also minimizes ARP (or ND) message
   processing on end-stations/hosts connected to the EVPN network.  A PE
   may learn the IP address associated with a MAC address in the control
   or management plane between the CE and the PE.  Or, it may learn this
   binding by snooping certain messages to or from a CE.  When a PE
   learns the IP address associated with a MAC address of a locally
   connected CE, it may advertise this address to other PEs by including
   it in the MAC/IP Advertisement route.  The IP address may be an IPv4
   address encoded using 4 octets or an IPv6 address encoded using
   16 octets.  For ARP and ND purposes, the IP Address Length field MUST
   be set to 32 for an IPv4 address or 128 for an IPv6 address.

   If there are multiple IP addresses associated with a MAC address,
   then multiple MAC/IP Advertisement routes MUST be generated, one for
   each IP address.  For instance, this may be the case when there are
   both an IPv4 and an IPv6 address associated with the same MAC address
   for dual-IP-stack scenarios.  When the IP address is dissociated with
   the MAC address, then the MAC/IP Advertisement route with that
   particular IP address MUST be withdrawn.

   Note that a MAC-only route can be advertised along with, but
   independent from, a MAC/IP route for scenarios where the MAC learning
   over an access network/node is done in the data plane and independent
   from ARP snooping that generates a MAC/IP route.  In such scenarios,
   when the ARP entry times out and causes the MAC/IP to be withdrawn,
   then the MAC information will not be lost.  In scenarios where the
   host MAC/IP is learned via the management or control plane, then the
   sender PE may only generate and advertise the MAC/IP route.  If the
   receiving PE receives both the MAC-only route and the MAC/IP route,
   then when it receives a withdraw message for the MAC/IP route, it
   MUST delete the corresponding entry from the ARP table but not the
   MAC entry from the MAC-VRF table, unless it receives a withdraw
   message for the MAC-only route.

   When a PE receives an ARP Request for an IP address from a CE, and if
   the PE has the MAC address binding for that IP address, the PE SHOULD
   perform ARP proxy by responding to the ARP Request.