Home > Networking Solutions > Converged and Hyperconverged Solutions > VxRail Networking Solutions > Guides > Dell EMC Networking SmartFabric Services Deployment with VxRail 4.7 > Move L3 uplinks to spine switches
Note: L2 uplinks are not applicable to this section. Only L3 uplinks are supported from the spine switches. L3 uplink configuration from leaf switches is covered in Configure L3 routed uplinks to the external network.
L3 uplinks to the external network can be moved from the leaf layer to the spine layer if desired. This section covers the migration process.
Figure 112 shows the existing L3 uplinks from the leafs to the external switches in purple. The new connections from the spines to the external network are shown in yellow. In this example, 40GbE connections are used from the external switches to the SmartFabric switches.
Note: In SmartFabric mode, the auto breakout feature will automatically break out the 100GbE interfaces on the Spine switches to 40GbE if connected to 40GbE ports on the external switches and Dell EMC supported cables and optics are used.
Figure 112. Existing and new L3 uplink connections to external switches
Figure 113 shows the autonomous system numbers (ASNs) and router IDs used for the external switches. The external switches share a common ASN. SmartFabric leafs share a common ASN, and SmartFabric spines share a common ASN as shown.
Figure 113. BGP ASNs and router IDs
In this example, ASN 65101 is used on both external switches. By default, all SmartFabric leaf switches use 65011, and all SmartFabric Spine switches use 65012.
The IP addresses shown in Figure 113 are loopback addresses used as BGP router IDs on the external network switches. Router IDs on the SFS switches are automatically configured from the SFS default private subnet address block, 172.16.0.0/16.
The existing point-to-point links used in this deployment are labeled A-D, and the new point-to-point links are labeled E-H in Figure 114.
Figure 114. Point-to-point connections
Each L3 uplink is a separate, point-to-point IP network. Table 13 details the links labeled in Figure 114. The IP addresses in the table are used in the switch configuration examples.
Table 13. L3 uplink IP addresses
Link label |
Source switch |
Source IP address |
Destination switch |
Destination IP address |
Network |
A |
External-A |
192.168.1.0 |
Leaf1A |
192.168.1.1 |
192.168.1.0/31 |
B |
External-A |
192.168.1.2 |
Leaf1B |
192.168.1.3 |
192.168.1.2/31 |
C |
External-B |
192.168.2.0 |
Leaf1A |
192.168.2.1 |
192.168.2.0/31 |
D |
External-B |
192.168.2.2 |
Leaf1B |
192.168.2.3 |
192.168.2.2/31 |
E |
External-A |
192.168.1.4 |
Spine1 |
192.168.1.5 |
192.168.1.4/31 |
F |
External-A |
192.168.1.6 |
Spine2 |
192.168.1.7 |
192.168.1.6/31 |
G |
External-B |
192.168.2.4 |
Spine1 |
192.168.2.5 |
192.168.2.4/31 |
H |
External-B |
192.168.2.6 |
Spine2 |
192.168.2.7 |
192.168.2.6/31 |
Table 14 shows the values entered in the OMNI plug-in in vCenter for this example. The following steps are run once for each uplink using the values from the table.
Table 14. Uplink configuration details
Link label |
E |
F |
G |
H |
L3 Type |
L3 Routed |
L3 Routed |
L3 Routed |
L3 Routed |
Uplink Name |
Spine01 to ExternalA |
Spine02 to ExternalA |
Spine01 to ExternalB |
Spine02 to ExternalB |
Switch Group |
Spine |
Spine |
Spine |
Spine |
Domain |
Fabric 1 |
Fabric 1 |
Fabric 1 |
Fabric 1 |
Spine Switch |
Spine01 |
Spine02 |
Spine01 |
Spine02 |
Interface |
Ethernet 1/1/27:1 |
Ethernet 1/1/27:1 |
Ethernet 1/1/28:1 |
Ethernet 1/1/28:1 |
Network Name |
SPN01 to ExtA |
SPN02 to ExtA |
SPN01 to ExtB |
SPN02 to ExtB |
Prefix Length |
31 |
31 |
31 |
31 |
IPv4 Address |
192.168.1.5 |
192.168.1.7 |
192.168.2.5 |
192.168.2.7 |
Protocol |
eBGP |
eBGP |
eBGP |
eBGP |
Router ID |
10.0.1.1 |
10.0.1.2 |
10.0.1.1 |
10.0.1.2 |
Policy ID/ Routing Policy Name |
eBGP SPN01 to ExtA |
eBGP SPN02 to ExtA |
eBGP SPN01 to ExtB |
eBGP SPN02 to ExtB |
Peer IPv4 Address |
192.168.1.4 |
192.168.1.6 |
192.168.2.4 |
192.168.2.6 |
Remote ASN |
65101 |
65101 |
65101 |
65101 |
In the OMNI plugin in vCenter, select the Service Instance in the left pane. In the right pane, click the Uplink tab. The existing uplinks from the leaf switches are displayed, as shown.
Figure 115. OMNI Uplink tab
To create uplinks from the spine switches, do the following:
Figure 116. Uplink Details page
Figure 117. Port Configuration page
Note: Policy ID and Routing Policy Name are text fields and do not have to match. However, the fields must be unique for each link, as shown in Table 14.
Figure 118. Network Configuration page
Repeat the steps in this section for the remaining three uplinks using the data from Table 14.
Once all four uplinks are configured, they are visible on the Uplink tab of the OMNI plugin. The new uplinks are outlined in red in Figure 119.
Figure 119. Uplinks from spines created
The external switches start at their configuration shown in Chapter 5 - Configure external switches for L3 connections. This section covers the configuration changes for both external switches.
Configure the interfaces for L3 connections to the SmartFabric spine switches. Interfaces 1/1/27 and 1/1/28 are used in this example.
Enter configuration mode with the configure terminal command.
For each interface, add a description and take the port out of L2 mode with the no switchport command. Assign an IP address to each interface per Table 13.
Optionally, allow the forwarding of jumbo frames with the MTU 9216 command. As a best practice, flow control settings remain at their factory defaults as shown.
External-A |
External-B |
configure terminal
interface ethernet1/1/27 description Spine1 no shutdown no switchport mtu 9216 ip address 192.168.1.4/31 flowcontrol receive on flowcontrol transmit off
interface ethernet1/1/28 description Spine2 no shutdown no switchport mtu 9216 ip address 192.168.1.6/31 flowcontrol receive on flowcontrol transmit off |
configure terminal
interface ethernet1/1/27 description Spine1 no shutdown no switchport mtu 9216 ip address 192.168.2.4/31 flowcontrol receive on flowcontrol transmit off
interface ethernet1/1/28 description Spine2 no shutdown no switchport mtu 9216 ip address 192.168.2.6/31 flowcontrol receive on flowcontrol transmit off |
Enter BGP configuration mode with the router bgp 65101 command. The external switches share the same ASN. For the router ID, use the address that was set for interface loopback0 in Chapter 5 - Configure external switches for L3 connections.
Configure the neighbor addresses per Table 13.
Configure the neighbor ASN. By default, all SmartFabric spine switches use 65012.
When the configuration is complete, exit configuration mode and save the configuration with the end and write memory commands.
External-A |
External-B |
router bgp 65101 router-id 10.0.2.1
neighbor 192.168.1.5 remote-as 65012 no shutdown
neighbor 192.168.1.7 remote-as 65012 no shutdown
end write memory |
router bgp 65101 router-id 10.0.2.2
neighbor 192.168.2.5 remote-as 65012 no shutdown
neighbor 192.168.2.7 remote-as 65012 no shutdown
end write memory |
Once the uplink interfaces are configured on the external switches and in the SFS GUI, connectivity can be verified using the switch CLI.
Note: The command output shown in the following commands is for the External-A switch. The output for External-B is similar.
Run the show ip bgp summary command to verify that BGP is up for each neighbor. When BGP is up, uptime is shown in the Up/Down column. The neighbors for External-A shown in the output below are Leaf1A, Leaf1B, Spine1, Spine2, and External-B.
Note: Leaf1A and Leaf1B can be disconnected from the external switches after validation is complete.
External-A# show ip bgp summary
BGP router identifier 10.0.2.1 local AS number 65101
Neighbor AS MsgRcvd MsgSent Up/Down State/Pfx
192.168.1.1 65011 3407 3404 07:10:58 8
192.168.1.3 65011 3409 3389 07:10:10 8
192.168.1.5 65012 76 66 00:13:36 6
192.168.1.7 65012 59 45 00:09:28 6
192.168.3.21 65101 5013 5019 3d:00:20:57 6
Run the show ip interface brief command to verify connected interfaces are up and IP addresses are configured correctly.
In the output below, interfaces 1/1/27 and 1/1/28 are connected to the spine switches.
Note: Unused interfaces have been removed from the output for brevity.
External-A# show ip interface brief
Interface Name IP-Address OK Method Status Protocol
================================================================================
Ethernet 1/1/1 unassigned YES unset up up
Ethernet 1/1/25 192.168.1.0/31 YES manual up up
Ethernet 1/1/26 192.168.1.2/31 YES manual up up
Ethernet 1/1/27 192.168.1.4/31 YES manual up up
Ethernet 1/1/28 192.168.1.6/31 YES manual up up
Ethernet 1/1/29 unassigned YES unset up up
Ethernet 1/1/30 unassigned YES unset up up
Management 1/1/1 100.67.198.28/24 YES manual up up
Vlan 1 unassigned YES unset up up
Vlan 1711 172.17.11.251/24 YES manual up up
Vlan 4000 192.168.3.20/31 YES manual up up
Vlan 4094 unassigned YES unset up up
Port-channel 1000 unassigned YES unset up up
Loopback 0 10.0.2.1/32 YES manual up up
Note: The command output shown in the following commands is for Spine1. The output for Spine2 is similar.
Run the show ip bgp summary command to verify that BGP is up for each neighbor. When BGP is up, uptime is shown in the Up/Down column. The neighbors for Spine1 shown in the output below are the four leaf switches (with automatically configured IP addresses in the 172.16.0.0/16 address range), and External-A and External-B.
Z9264F-Spine1# show ip bgp summary
BGP router identifier 172.16.128.3 local AS number 65012
Neighbor AS MsgRcvd MsgSent Up/Down State/Pfx
172.16.0.2 65011 103 110 01:19:12 23
172.16.0.4 65011 105 111 01:18:43 23
172.16.0.6 65011 126 112 01:18:24 8
172.16.0.16 65011 123 112 01:18:20 8
192.168.1.4 65101 22 29 00:15:59 15
192.168.2.4 65101 16 17 00:10:51 15
Run the show ip interface brief command to verify connected interfaces are up, and IP addresses are configured correctly.
In the output below, interfaces 1/1/27:1 and 1/1/28:1 are connected to the external switches.
Note: Unused interfaces have been removed from the output for brevity.
Z9264F-Spine1# show ip interface brief
Interface Name IP-Address OK Method Status Protocol
================================================================================
Ethernet 1/1/13 unassigned YES unset up up
Ethernet 1/1/14 unassigned YES unset up up
Ethernet 1/1/15 unassigned YES unset up up
Ethernet 1/1/16 unassigned YES unset up up
Ethernet 1/1/27:1 192.168.1.5/31 YES manual up up
Ethernet 1/1/28:1 192.168.2.5/31 YES manual up up
Management 1/1/1 100.67.198.36/24 YES manual up up
Vlan 1 unassigned YES unset up up
Vlan 4000 unassigned YES unset up up
Vlan 4001 172.16.0.11/31 YES manual up up
Vlan 4002 172.16.0.13/31 YES manual up up
Vlan 4003 172.16.0.9/31 YES manual up up
Vlan 4004 172.16.0.17/31 YES manual up up
Port-channel 96 unassigned YES unset up up
Port-channel 97 unassigned YES unset up up
Port-channel 98 unassigned YES unset up up
Port-channel 99 unassigned YES unset up up
Loopback 1 172.16.128.4/32 YES manual up up
After the new uplinks from the spines are validated, the original uplinks from the leafs may be removed as follows: