For multirack deployments, BGP EVPN with VXLAN stretches virtual (overlay) Layer 2 networks across a physical (underlay) Layer 3 leaf-spine fabric. This architecture allows for the scalability of Layer 3 networks with the mobility benefits of a Layer 2 network. For example, a VM or physical host can be moved from one rack to another without changing its IP address and gateway information.
EVPN uses BGP to exchange endpoint MAC and IP address information between VXLAN tunnel endpoints (VTEPs). In this topology, each pair of leaf switches in a rack is one logical VTEP. Packets destined to a remote device are sent to the local VTEP as the first hop. The packet is encapsulated with a VXLAN header that includes the Virtual Network Identifier (VNI) and the IP address of the destination host. See the figure below.
The destination VTEP extracts the original packet and routes it to the final destination. VXLAN encapsulation adds approximately 50 bytes of overhead to each Ethernet frame. As a result, all switches in the underlay (physical) network must be configured to support an MTU of at least 1600 bytes on all participating interfaces.
Anycast gateways with the same IP address are configured on each leaf pair. The anycast gateway IP address is set as the default gateway for all hosts and VMs on that virtual network. Hosts and VMs with anycast gateways use the same gateway information while behind different leaf pairs. This configuration replaces Virtual Router Redundancy Protocol (VRRP) and enables hosts and VMs to migrate from one leaf pair to another without changing the network configuration.