Your Browser is Out of Date

Nytro.ai uses technology that works best in other browsers.
For a full experience use one of the browsers below
VMware Cloud Foundation on VxRail Multirack Deployment Using BGP EVPN - Part 2 of 2
Introduction Hardware Overview Network Transport Topology Planning and Preparation Configure and Verify the Underlay Network
Configure leaf switch underlay networking
Configure leaf switch NSX-T overlay networking
Configure spine switches
Verify establishment of BGP between leaf and spine switches
Verify BGP EVPN and VXLAN between leaf switches
Create a VxRail Virtual Infrastructure Workload Domain Configure NSX-T North-South Connectivity Validate Connectivity Between Virtual Machines Validated Components Technical Resources Fabric Design Center Support and Feedback Attachments

Configure leaf switch underlay networking

Home > Networking Solutions > Converged and Hyperconverged Solutions > VxRail Networking Solutions > Guides > VMware Cloud Foundation on VxRail Multirack Deployment Using BGP EVPN - Part 2 of 2 > Configure leaf switch underlay networking

Configure leaf switch underlay networking

  • This chapter details the configuration for S5248F-ON switch with the hostname sfo01-Leaf01a, shown as the left switch in Figure 17. Virtual networks 1641 and 3939 are shown in the diagram as an example. All the required virtual networks are created during the switch configuration. Configuration differences for Leaf switch 1b, 2a, and 2b are noted in Switch settings. These commands should be entered in the order shown.

    Note: This deployment uses four leaf switches. All four leaf switch configuration files are provided as annotated text file attachments to this document. Attachments describes how to access .pdf attachments. All switches start at their factory default settings, as described in Factory default configuration.

    Figure 17.     Rack 1, leaf switch diagram

    Note: Some of the steps below may already be done if all configuration steps from the VCF on VxRail Multirack Deployment using BGP EVPN: Creating a management domain with NSX-V (part 1 of 2) deployment guide were followed. All of the steps are included here to ensure completion.

    1. Configure general switch settings, including management and NTP source.

    OS10# configure terminal

    OS10(config)# interface mgmt 1/1/1

    OS10(conf-if-ma-1/1/1)# no ip address dhcp

    OS10(conf-if-ma-1/1/1)# ip address 100.67.198.32/24

    OS10(conf-if-ma-1/1/1)# exit

    OS10(config)# management route 100.67.0.0/16 managementethernet

    OS10(config)# hostname sfo01-Leaf01A

    sfo01-Leaf01A(config)# ntp server 100.67.10.20

    sfo01-Leaf01A(config)# bfd enable

    sfo01-Leaf01A(config)# ipv6 mld snooping enable

    1. Configure a loopback interface for the Router ID using the following command:

    sfo01-Leaf01A(config)# interface loopback 0

    sfo01-Leaf01A(conf-if-lo-0)# description Router-ID

    sfo01-Leaf01A(conf-if-lo-0)# no shutdown

    sfo01-Leaf01A(conf-if-lo-0)# ip address 10.0.2.1/32

    sfo01-Leaf01A(conf-if-lo-0)# exit

    1. Configure a loopback interface for NVE.

    sfo01-Leaf01A(config)# interface loopback 2

    sfo01-Leaf01A(conf-if-lo-2)# description nve_loopback

    sfo01-Leaf01A(conf-if-lo-2)# no shutdown

    sfo01-Leaf01A(conf-if-lo-2)# ip address 10.222.222.1/32

    sfo01-Leaf01A(conf-if-lo-2)# exit

    1. Configure the loopback interface for the VXLAN source tunnel interface.

    sfo01-Leaf01A(config)# nve

    sfo01-Leaf01A(config-nve)# source-interface loopback2

    sfo01-Leaf01A(config-nve)# exit

    1. Using the following commands, configure the VXLAN virtual networks:

    sfo01-Leaf01A(config)# virtual-network 1641

    sfo01-Leaf01A(config-vn)# vxlan-vni 1641

    sfo01-Leaf01A(config-vn)# exit

    sfo01-Leaf01A(config)# virtual-network 1642

    sfo01-Leaf01A(config-vn)# vxlan-vni 1642

    sfo01-Leaf01A(config-vn)# exit

    sfo01-Leaf01A(config)# virtual-network 1643

    sfo01-Leaf01A(config-vn)# vxlan-vni 1643

    sfo01-Leaf01A(config-vn)# exit

    sfo01-Leaf01A(config)# virtual-network 3939

    sfo01-Leaf01A(config-vn)# vxlan-vni 3939

    sfo01-Leaf01A(config-vn)# exit

    1. Assign the VLAN member interfaces to virtual networks.

    sfo01-Leaf01A(config)# interface vlan1641

    sfo01-Leaf01A(config-if-vl-1641)# description sfo-w02-mgmt

    sfo01-Leaf01A(config-if-vl-1641)# virtual-network 1641

    sfo01-Leaf01A(config-if-vl-1641)# no shutdown

    sfo01-Leaf01A(config-if-vl-1641)# mtu 9216

    sfo01-Leaf01A(config-if-vl-1641)# exit

    sfo01-Leaf01A(config)# interface vlan1642

    sfo01-Leaf01A(config-if-vl-1642)# virtual-network 1642

    sfo01-Leaf01A(config-if-vl-1642)# description sfo-w02-vmotion

    sfo01-Leaf01A(config-if-vl-1642)# no shutdown

    sfo01-Leaf01A(config-if-vl-1642)# mtu 9216

    sfo01-Leaf01A(config-if-vl-1642)# exit

    sfo01-Leaf01A(config)# interface vlan1643

    sfo01-Leaf01A(config-if-vl-1643)# virtual-network 1643

    sfo01-Leaf01A(config-if-vl-1643)# description sfo-w02-san

    sfo01-Leaf01A(config-if-vl-1643)# no shutdown

    sfo01-Leaf01A(config-if-vl-1643)# mtu 9216

    sfo01-Leaf01A(config-if-vl-1643)# exit

    sfo01-Leaf01A(config)# interface vlan3939

    sfo01-Leaf01A(config-if-vl-3939)# description vxrail-mgmt

    sfo01-Leaf01A(config-if-vl-3939)# virtual-network 3939

    sfo01-Leaf01A(config-if-vl-3939)# ipv6 mld snooping querier

    sfo01-Leaf01A(config-if-vl-3939)# no shutdown

    sfo01-Leaf01A(config-if-vl-3939)# mtu 9216

    sfo01-Leaf01A(config-if-vl-3939)# exit

    Note: Enable the ipv6 mld snooping querier on any logical network that requires multicast support.

    1. Configure access ports as VLAN members for a switch-scoped VLAN-to-VNI mapping. VLAN 1641 is untagged to accommodate ESXi management, which is configured by default as untagged on the management VMkernel. The remaining VLANs are tagged to support the tagging from the default VxRail created vDS.

    sfo01-Leaf01A(config)# interface ethernet1/1/3

    sfo01-Leaf01A(conf-if-eth1/1/1)# description sfo01w02vxrail01

    sfo01-Leaf01A(conf-if-eth1/1/1)# no shutdown

    sfo01-Leaf01A(conf-if-eth1/1/1)# switchport mode trunk

    sfo01-Leaf01A(conf-if-eth1/1/1)# switchport access vlan 1641

    sfo01-Leaf01A(conf-if-eth1/1/1)# switchport trunk allowed vlan 1642-1643,3939

    sfo01-Leaf01A(conf-if-eth1/1/1)# mtu 9216

    sfo01-Leaf01A(conf-if-eth1/1/1)# spanning-tree port type edge

    sfo01-Leaf01A(conf-if-eth1/1/1)# flowcontrol receive on

    sfo01-Leaf01A(conf-if-eth1/1/1)# flowcontrol transmit off

    sfo01-Leaf01A(conf-if-eth1/1/1)# exit

    sfo01-Leaf01A(config)# interface ethernet1/1/4

    sfo01-Leaf01A(conf-if-eth1/1/2)# description sfo01w02vxrail02

    sfo01-Leaf01A(conf-if-eth1/1/2)# no shutdown

    sfo01-Leaf01A(conf-if-eth1/1/2)# switchport mode trunk

    sfo01-Leaf01A(conf-if-eth1/1/2)# switchport access vlan 1641

    sfo01-Leaf01A(conf-if-eth1/1/2)# switchport trunk allowed vlan 1642-1643,3939

    sfo01-Leaf01A(conf-if-eth1/1/2)# mtu 9216

    sfo01-Leaf01A(conf-if-eth1/1/2)# spanning-tree port type edge

    sfo01-Leaf01A(conf-if-eth1/1/2)# flowcontrol receive on

    sfo01-Leaf01A(conf-if-eth1/1/2)# flowcontrol transmit off

    sfo01-Leaf01A(conf-if-eth1/1/2)# exit

    1. Enter the following commands to configure upstream network-facing ports:

    sfo01-Leaf01A(config)# interface ethernet1/1/53

    sfo01-Leaf01A(conf-if-eth1/1/53)# description sfo01-spine01

    sfo01-Leaf01A(conf-if-eth1/1/53)# no shutdown

    sfo01-Leaf01A(conf-if-eth1/1/53)# no switchport

    sfo01-Leaf01A(conf-if-eth1/1/53)# mtu 9216

    sfo01-Leaf01A(conf-if-eth1/1/53)# ip address 192.168.1.1/31

    sfo01-Leaf01A(conf-if-eth1/1/53)# exit

    sfo01-Leaf01A(config)# interface ethernet1/1/54

    sfo01-Leaf01A(conf-if-eth1/1/54)# description sfo01-spine02

    sfo01-Leaf01A(conf-if-eth1/1/54)# no shutdown

    sfo01-Leaf01A(conf-if-eth1/1/54)# no switchport

    sfo01-Leaf01A(conf-if-eth1/1/54)# mtu 9216

    sfo01-Leaf01A(conf-if-eth1/1/54)# ip address 192.168.2.1/31

    sfo01-Leaf01A(conf-if-eth1/1/54)# exit

    1. Add a route map. This example route map is used to illustrate how to allow IP traffic to be passed on the switch.

    sfo01-Leaf01A(config)# ip prefix-list spine-leaf seq 10 permit 10.0.2.0/24 ge 32

    sfo01-Leaf01A(config)# ip prefix-list spine-leaf seq 20 permit 10.2.2.0/24 ge 32

    sfo01-Leaf01A(config)# ip prefix-list spine-leaf seq 30 permit 10.222.222.0/24 ge 32

    sfo01-Leaf01A(config)# route-map spine-leaf permit 10

    sfo01-Leaf01A(config-route-map)# match ip address prefix-list spine-leaf

    sfo01-Leaf01A(config-route-map)# exit

    1. Configure eBGP

    sfo01-Leaf01A(config)# router bgp 65101

    sfo01-Leaf01A(config-router-bgp-65101)# router-id 10.0.2.1

    sfo01-Leaf01A(config-router-bgp-65101)# bfd all-neighbors interval 200 min_rx 200 multiplier 3 role active

    sfo01-Leaf01A(config-router-bgp-65101)# address-family ipv4 unicast

    sfo01-Leaf01A(config-router-bgpv4-af)# redistribute connected route-map spine-leaf

    sfo01-Leaf01A(config-router-bgpv4-af)# exit

    sfo01-Leaf01A(config-router-bgp-65101)# bestpath as-path multipath-relax

    sfo01-Leaf01A(config-router-bgp-65101)# maximum-paths ebgp 2

    Note: If more than two ESGs are being used, update the maximum-paths ebgp value accordingly.

    1. Configure eBGP for the IPv4 point-to-point peering using the following commands:

    sfo01-Leaf01A(config-router-bgp-65101)# neighbor 192.168.1.0

    sfo01-Leaf01A(config-router-neighbor)# advertisement-interval 5

    sfo01-Leaf01A(config-router-neighbor)# bfd

    sfo01-Leaf01A(config-router-neighbor)# fall-over

    sfo01-Leaf01A(config-router-neighbor)# remote-as 65100

    sfo01-Leaf01A(config-router-neighbor)# no shutdown

    sfo01-Leaf01A(config-router-neighbor)# address-family ipv4 unicast

    sfo01-Leaf01A(config-router-neighbor-af)# exit

    sfo01-Leaf01A(config-router-neighbor)# exit

    sfo01-Leaf01A(config-router-bgp-65101)# neighbor 192.168.2.0

    sfo01-Leaf01A(config-router-neighbor)# advertisement-interval 5

    sfo01-Leaf01A(config-router-neighbor)# bfd

    sfo01-Leaf01A(config-router-neighbor)# fall-over

    sfo01-Leaf01A(config-router-neighbor)# remote-as 65100

    sfo01-Leaf01A(config-router-neighbor)# no shutdown

    sfo01-Leaf01A(config-router-neighbor)# address-family ipv4 unicast

    sfo01-Leaf01A(config-router-neighbor-af)# exit

    sfo01-Leaf01A(config-router-neighbor)# exit

    sfo01-Leaf01A(config-router-bgp-65101)# exit

    1. Configure a loopback interface for BGP EVPN peering.

    sfo01-Leaf01A(config)# interface loopback 1

    sfo01-Leaf01A(conf-if-lo-1)# description evpn_loopback

    sfo01-Leaf01A(conf-if-lo-1)# no shutdown

    sfo01-Leaf01A(conf-if-lo-1)# ip address 10.2.2.1/32

    sfo01-Leaf01A(conf-if-lo-1)# exit

    1. Enter the following commands to configure BGP EVPN peering:

    sfo01-Leaf01A(config)# router bgp 65101

    sfo01-Leaf01A(config-router-bgp-65101)# neighbor 10.2.1.1

    sfo01-Leaf01A(config-router-neighbor)# remote-as 65100

    sfo01-Leaf01A(config-router-neighbor)# ebgp-multihop 2

    sfo01-Leaf01A(config-router-neighbor)# send-community extended

    sfo01-Leaf01A(config-router-neighbor)# update-source loopback1

    sfo01-Leaf01A(config-router-neighbor)# no shutdown

    sfo01-Leaf01A(config-router-neighbor)# address-family ipv4 unicast

    sfo01-Leaf01A(config-router-neighbor-af)# no activate

    sfo01-Leaf01A(config-router-neighbor-af)# exit

    sfo01-Leaf01A(config-router-neighbor)# address-family l2vpn evpn

    sfo01-Leaf01A(config-router-neighbor-af)# activate

    sfo01-Leaf01A(config-router-neighbor-af)# exit

    sfo01-Leaf01A(config-router-neighbor)# exit

    sfo01-Leaf01A(config-router-bgp-65101)# neighbor 10.2.1.2

    sfo01-Leaf01A(config-router-neighbor)# remote-as 65100

    sfo01-Leaf01A(config-router-neighbor)# ebgp-multihop 2

    sfo01-Leaf01A(config-router-neighbor)# send-community extended

    sfo01-Leaf01A(config-router-neighbor)# update-source loopback1

    sfo01-Leaf01A(config-router-neighbor)# no shutdown

    sfo01-Leaf01A(config-router-neighbor)# address-family ipv4 unicast

    sfo01-Leaf01A(config-router-neighbor-af)# no activate

    sfo01-Leaf01A(config-router-neighbor-af)# exit

    sfo01-Leaf01A(config-router-neighbor)# address-family l2vpn evpn

    sfo01-Leaf01A(config-router-neighbor-af)# activate

    sfo01-Leaf01A(config-router-neighbor-af)# exit

    sfo01-Leaf01A(config-router-neighbor)# exit

    sfo01-Leaf01A(config-router-bgp-65101)# exit

    1. Configure EVPN

    sfo01-Leaf01A(config)# evpn

    sfo01-Leaf01A(config-evpn)# evi 1641

    sfo01-Leaf01A(config-evpn-evi-1641)# vni 1641

    sfo01-Leaf01A(config-evpn-evi-1641)# rd 10.222.222.1:1641

    sfo01-Leaf01A(config-evpn-evi-1641)# route-target 1641:1641 both

    sfo01-Leaf01A(config-evpn-evi-1641)# exit

    sfo01-Leaf01A(config-evpn)# evi 1642

    sfo01-Leaf01A(config-evpn-evi-1642)# vni 1642

    sfo01-Leaf01A(config-evpn-evi-1642)# rd 10.222.222.1:1642

    sfo01-Leaf01A(config-evpn-evi-1642)# route-target 1642:1642 both

    sfo01-Leaf01A(config-evpn-evi-1642)# exit

    sfo01-Leaf01A(config-evpn)# evi 1643

    sfo01-Leaf01A(config-evpn-evi-1643)# vni 1643

    sfo01-Leaf01A(config-evpn-evi-1643)# rd 10.222.222.1:1643

    sfo01-Leaf01A(config-evpn-evi-1643)# route-target 1643:1643 both

    sfo01-Leaf01A(config-evpn-evi-1643)# exit

    1. Configure the dedicated L3 underlay path to reach a VLT peer in the event of network failure:

    sfo01-Leaf01A(config)# interface vlan4000

    sfo01-Leaf01A(config-if-vl-4000)# no shutdown

    sfo01-Leaf01A(config-if-vl-4000)# mtu 9216

    sfo01-Leaf01A(config-if-vl-4000)# ip address 192.168.3.0/31

    sfo01-Leaf01A(config-if-vl-4000)# exit

    1. Configure the VLTi member links.

    sfo01-Leaf01A(config)# interface range ethernet1/1/49-1/1/52

    sfo01-Leaf01A(conf-range-eth1/1/49-1/1/52)# description VLTi

    sfo01-Leaf01A(conf-range-eth1/1/49-1/1/52)# no shutdown

    sfo01-Leaf01A(conf-range-eth1/1/49-1/1/52)# no switchport

    sfo01-Leaf01A(conf-range-eth1/1/49-1/1/52)# exit

    1. Configure the VLT domain.

    sfo01-Leaf01A(config)# vlt-domain 1

    sfo01-Leaf01A(conf-vlt-1)# backup destination 100.67.198.31

    sfo01-Leaf01A(conf-vlt-1)# discovery-interface ethernet1/1/49-1/1/52

    sfo01-Leaf01A(conf-vlt-1)# peer-routing

    sfo01-Leaf01A(conf-vlt-1)# vlt-mac 00:00:01:02:03:01

    sfo01-Leaf01A(conf-vlt-1)# exit

    1. Configure the iBGP IPv4 peering between the VLT peers.

    sfo01-Leaf01A(config)# router bgp 65101

    sfo01-Leaf01A(config-router-bgp-65101)# neighbor 192.168.3.1

    sfo01-Leaf01A(config-router-neighbor)# remote-as 65101

    sfo01-Leaf01A(config-router-neighbor)# no shutdown

    sfo01-Leaf01A(config-router-neighbor)# exit

    1. Create a tenant VRF.

    Note: An OS10 best practice is to isolate any virtual network traffic in a non-default VRF.

    sfo01-Leaf01A(config)# ip vrf tenant1

    sfo01-Leaf01A(conf-vrf)# exit

    1. Configure the anycast gateway MAC address.

    sfo01-Leaf01A(config)# ip virtual-router mac-address 00:01:01:01:01:01

    1. Configure routing on virtual networks.

    sfo01-Leaf01A(config)# interface virtual-network1641

    sfo01-Leaf01A(conf-if-vn-1641)# no shutdown

    sfo01-Leaf01A(conf-if-vn-1641)# mtu 9216

    sfo01-Leaf01A(conf-if-vn-1641)# ip vrf forwarding tenant1

    sfo01-Leaf01A(conf-if-vn-1641)# ip address 172.16.41.252/24

    sfo01-Leaf01A(conf-if-vn-1641)# ip virtual-router address 172.16.41.253

    sfo01-Leaf01A(conf-if-vn-1641)# exit

    sfo01-Leaf01A(config)# interface virtual-network1642

    sfo01-Leaf01A(conf-if-vn-1642)# no shutdown

    sfo01-Leaf01A(conf-if-vn-1642)# mtu 9216

    sfo01-Leaf01A(conf-if-vn-1642)# ip vrf forwarding tenant1

    sfo01-Leaf01A(conf-if-vn-1642)# ip address 172.16.42.252/24

    sfo01-Leaf01A(conf-if-vn-1642)# ip virtual-router address 172.16.42.253

    sfo01-Leaf01A(conf-if-vn-1642)# exit

    sfo01-Leaf01A(config)# interface virtual-network1643

    sfo01-Leaf01A(conf-if-vn-1643)# no shutdown

    sfo01-Leaf01A(conf-if-vn-1643)# mtu 9216

    sfo01-Leaf01A(conf-if-vn-1643)# ip vrf forwarding tenant1

    sfo01-Leaf01A(conf-if-vn-1643)# ip address 172.16.43.252/24

    sfo01-Leaf01A(conf-if-vn-1643)# ip virtual-router address 172.16.43.253

    sfo01-Leaf01A(conf-if-vn-1643)# exit

    sfo01-Leaf01A(config)# interface virtual-network3939

    sfo01-Leaf01A(conf-if-vn-3939)# no shutdown

    sfo01-Leaf01A(conf-if-vn-3939)# ip vrf forwarding tenant1

    sfo01-Leaf01A(conf-if-vn-3939)# exit