Uplinks to the existing network may be configured as L2, L3 routed, or L3 VLAN. This section covers L3 routed uplinks.
Connections, port numbers, and networks used for external management are shown in the figure below. The External Management VLAN is VLAN 1911 on the external switches and is VLAN 1811 on the SmartFabric switches.
Point-to-point IP networks
The point-to-point links used in this deployment are labeled A-E in Point-to-point connections.
Each L3 uplink is a separate, point-to-point IP network. L3 routed uplink IP addresses details the links labeled in Point-to-point connections. The IP addresses in the table below are used in the switch configuration examples.
Link label | Source switch | Source IP address | Destination switch | Destination IP address | Network |
A | External-A | 192.168.1.0 | Leaf1A | 192.168.1.1 | 192.168.1.0/31 |
B | External-A | 192.168.1.2 | Leaf1B | 192.168.1.3 | 192.168.1.2/31 |
C | External-B | 192.168.2.0 | Leaf1A | 192.168.2.1 | 192.168.2.0/31 |
D | External-B | 192.168.2.2 | Leaf1B | 192.168.2.3 | 192.168.2.2/31 |
E | External-A | 192.168.3.20 | External-B | 192.168.3.21 | 192.168.3.20/31 |
BGP example
This section covers the L3 routed uplink configuration with BGP.
BGP ASNs and router IDs
BGP ASNs and router IDs shows the autonomous system numbers (ASNs) and router IDs used for the external switches and SFS leaf switches in this example. External switches share a common ASN, and all SFS leaf switches share a common ASN.
In this example, ASN 65101 is used on both external switches. SFS leaf switches use ASN 65011 by default for all leafs in the fabric.
The IP addresses shown on the external network switches in BGP ASNs and router IDs are loopback addresses used as BGP router IDs. On the SmartFabric switches, BGP router IDs are automatically configured from the SFS default private subnet address block, 172.16.0.0/16.
Configure L3 routed uplinks with BGP in SFS
The following table shows the values entered in the SFS web UI to configure the L3 uplinks for this example. The steps below the table are run once for each uplink using the values in the table.
Field name | Leaf1A-to-External-A | Leaf1A-to-External-B | Leaf1B-to-External-A | Leaf1B-to-External-B |
Uplink Port Type | L3 | L3 | L3 | L3 |
L3 Type | L3 Routed Interface | L3 Routed Interface | L3 Routed Interface | L3 Routed Interface |
Uplink Name | Leaf1A-ExtA | Leaf1A-ExtB | Leaf1B-ExtA | Leaf1B-ExtB |
Switch Group | Leaf | Leaf | Leaf | Leaf |
Rack | Rack 1 | Rack 1 | Rack 1 | Rack 1 |
Node | Leaf1A | Leaf1A | Leaf1B | Leaf1B |
Interface | Ethernet 1/1/53 | Ethernet 1/1/54 | Ethernet 1/1/53 | Ethernet 1/1/54 |
Network Name | Leaf1A-ExtA | Leaf1A-ExtB | Leaf1B-ExtA | Leaf1B-ExtB |
Prefix Length | 31 | 31 | 31 | 31 |
IP Address | 192.168.1.1 | 192.168.2.1 | 192.168.1.3 | 192.168.2.3 |
Routing Protocol | eBGP | eBGP | eBGP | eBGP |
Policy Name | Leaf1A-ExtA | Leaf1A-ExtB | Leaf1B-ExtA | Leaf1B-ExtB |
Peer Interface IP Address | 192.168.1.0 | 192.168.2.0 | 192.168.1.2 | 192.168.2.2 |
Peer ASN | 65101 | 65101 | 65101 | 65101 |
BFD Neighbor | Enable | Enable | Enable | Enable |
Fabric Interconnect | unchecked | unchecked | unchecked | unchecked |
To configure L3 routed uplinks with BGP, do the following using the data from the L3 uplink configuration details with BGP table above.
- In the SFS UI, select the Uplink tab.
- Under Uplink, click CREATE.
- On the Uplink Details page:
- Set Uplink Port Type to L3.
- Next to L3 Type, select L3 Routed Interface.
- Enter a unique uplink Name.
- Click NEXT.
- On the Port Configuration page:
- Leave the Switch Group set to Leaf.
- Next to Select Rack to Create Uplink on, select the rack that contains the switches with the uplinks. In this example, Rack 1 is selected.
- Next to Node, select the first leaf, Leaf1A in this example.
- In the Interface Selection list, select the interface that will be used for the uplink. In this example, interface 1/1/53 is selected.
- Click NEXT.
- On the Network Configuration page:
- Enter a unique network profile Name.
- Enter the Prefix Length and IP Address of the interface.
- Select the Routing Protocol, eBGP.
- Enter a unique Policy Name.
- Enter the Peer Interface IP Address and Peer ASN.
- Next to BFD Neighbor, select the Enable or Disable button. In this example, it is set to Enable. Note: Dell Technologies recommends that you enable Bidirectional Forwarding Detection (BFD) if the external switches support it. BFD rapidly detects communication failures between two adjacent routers.
- Leave the Fabric Interconnect box unchecked.
- Click FINISH. The Success message displays. Note: Creating the first uplink also sets the preferred leader flag on certain switches in the fabric, if not already configured. Additional Success messages indicating the preferred leader is set may display. The preferred leader setting is covered in Verify preferred leader setting before fabric expansion.
- Repeat the steps in this section for the remaining three uplinks using the data from the preceding table. After configuration, the uplinks are listed on the Uplink page of the SFS UI, as shown in the following figure. The Topology tab displays as shown in the following figure:
Static route example
This section shows L3 routed uplink configuration with a static route.
Configure L3 uplinks with a static route in SFS
The following table shows the values entered in the SFS UI to configure the L3 uplinks for this example. The steps below the table are run once for each uplink using the values from the table.
Field name | Leaf1A-to-External-A | Leaf1A-to-External-B | Leaf1B-to-External-A | Leaf1B-to-External-B |
Uplink Port Type | L3 | L3 | L3 | L3 |
L3 Type | L3 Routed Interface | L3 Routed Interface | L3 Routed Interface | L3 Routed Interface |
Uplink Name | Leaf1A-ExtA | Leaf1A-ExtB | Leaf1B-ExtA | Leaf1B-ExtB |
Switch Group | Leaf | Leaf | Leaf | Leaf |
Rack | Rack 1 | Rack 1 | Rack 1 | Rack 1 |
Node | Leaf1A | Leaf1A | Leaf1B | Leaf1B |
Interface | Ethernet 1/1/53 | Ethernet 1/1/54 | Ethernet 1/1/53 | Ethernet 1/1/54 |
Network Name | Leaf1A-ExtA | Leaf1A-ExtB | Leaf1B-ExtA | Leaf1B-ExtB |
Prefix Length | 31 | 31 | 31 | 31 |
IP Address | 192.168.1.1 | 192.168.2.1 | 192.168.1.3 | 192.168.2.3 |
Routing Protocol | Static Route | Static Route | Static Route | Static Route |
Policy Name | Leaf1A-ExtA | Leaf1A-ExtB | Leaf1B-ExtA | Leaf1B-ExtB |
Network Address | 172.19.11.0 | 172.19.11.0 | 172.19.11.0 | 172.19.11.0 |
Prefix Length | 24 | 24 | 24 | 24 |
Next Hop IP Address | 192.168.1.0 | 192.168.2.0 | 192.168.1.2 | 192.168.2.2 |
Fabric Interconnect | unchecked | unchecked | unchecked | unchecked |
To configure L3 routed uplinks with a static route, perform the following steps:
- In the SFS UI, select the Uplink tab.
- Under Uplink, click CREATE.
- On the Uplink Details page:
- Set Uplink Port Type to L3.
- Next to L3 Type, select L3 Routed Interface.
- Enter a unique Name.
- Click NEXT.
- On the Port Configuration page:
- Leave the Switch Group set to Leaf.
- Next to Select Rack to Create Uplink on, select the rack that contains the switches with the uplinks. In this example, Rack 1 is selected.
- Next to Node, select the first leaf, Leaf1A in this example.
- In the Interface Selection list, select the interface that will be used for the uplink. In this example, interface 1/1/53 is selected.
- Click NEXT.
- On the Network Configuration page:
- Enter a unique Name.
- Enter the Prefix Length and IP Address of the interface.
- Select the Routing Protocol, Static Route.
- Enter a unique Policy Name.
- Enter the destination Network Address and Prefix Length. This is the external network that contains DNS and NTP services, 172.19.11.0/24, in this example.
- Enter the Next Hop IP Address. This is the IP address of the connected interface on the external switch.
- Leave the Fabric Interconnect box unchecked.
- Click Finish. A Success message displays. Note: Creating the first uplink also sets the preferred leader flag on certain switches in the fabric, if not already configured. Other Success messages indicating the preferred leader is set may show. The preferred leader setting is covered later in this guide.
- Repeat the steps in this section for the remaining three uplinks using the data from the preceding table. After configuration, the uplinks are listed on the Uplink page of the SFS UI, as shown. The Topology tab displays as shown in the figure below.
Configure external switches for L3 connections
This section shows example configurations for both external switches for L3 routed connections to the SmartFabric.
General settings
Configure the hostname, OOB management IP address, and management route.
External-A | External-B |
|
|
Configure VLANs
VLAN 1911 represents a preexisting management VLAN on the external network. DNS and NTP services are on this VLAN. Assign a unique IP address to the VLAN on each switch.
Configure VRRP to provide gateway redundancy. Set the VRRP priority. The switch with the highest priority value becomes the master VRRP router. Assign the same virtual address to both switches.
External-A | External-B |
|
|
Configure interfaces
Configure the interfaces for connections to the SFS switches. Ports 1/1/13 and 1/1/14 are configured as L3 interfaces. The IP addresses used are from L3 routed uplink IP addresses. Optionally, allow the forwarding of jumbo frames with the MTU 9216 command. As a best practice, flow control settings remain at their factory defaults as shown.
In this example, VLT port channel 1 connects to the DNS/NTP server. It is on VLAN 1911, which represents the preexisting management VLAN, and the port channel is configured as a spanning tree edge port.
Interface 1/1/1 on each external switch is configured in VLT port channel 1 for connections to the DNS/NTP server. Port-channel 1 is set as an LACP port channel with the channel-group 1 mode active command.
External-A | External-B |
|
|
Configure VLT
This example uses interfaces 1/1/11 and 1/1/12 for the VLTi. Remove each interface from L2 mode with the no switchport command. As a best practice, flow control settings remain at their factory defaults, as shown.
Create the VLT domain. The backup destination is the OOB management IP address of the VLT peer switch. Configure the interfaces used as the VLTi with the discovery-interface command.
As a best practice, use the vlt-mac command to manually configure the same VLT MAC address on both the VLT peer switches. This improves VLT convergence time when a switch is reloaded.
If you do not configure a VLT MAC address, the MAC address of the primary peer is used as the VLT MAC address on both switches.
External-A | External-B |
|
|
Configure BGP
Configure a loopback interface to use for the BGP router ID.
Enable BFD.
Configure the BGP ASN with the router bgp command. The external switches share the same ASN. Use the address that was set for interface loopback0 as the router ID.
Use the address-family ipv4 unicast and redistribute connected commands to redistribute IPv4 routes from physically connected interfaces.
Configure the neighbor IP addresses, ASNs and enable BFD.
VLAN 4000 is used for the iBGP connection between the external switches. VLAN4000 IP addresses are configured as in L3 routed uplink IP addresses.
When the configuration is complete, exit configuration mode and save the configuration with the end and write memory commands.
External-A | External-B |
|
|
Configure static routes
Configure two routes to the external management network. This is 172.18.11.0/24, one to the connected IP address of Leaf1A, and one to Leaf1B.
When the configuration is complete, exit configuration mode and save the configuration with the end and write memory commands.
External-A | External-B |
|
|
Validate BGP example
Now that the uplink interfaces are configured on the external switches and on the SFS leaf switches, connectivity can be verified using the switch CLI.
Show command output on External-A (BGP example)
Run the show ip bgp summary command to verify that BGP is up for each neighbor. When BGP is up, uptime is shown in the Up/Down
column. The neighbors for External-A shown in the output below are Leaf1A, Leaf1B, and External-B.
External-A# show ip bgp summary
BGP router identifier 10.0.2.1 local AS number 65101
Neighbor AS MsgRcvd MsgSent Up/Down State/Pfx
192.168.1.1 65011 1327 1316 19:09:00 4
192.168.1.3 65011 1325 1324 19:09:00 4
192.168.3.21 65101 1319 1315 19:01:18 5
Run the show ip interface brief command to verify connected interfaces are up, and IP addresses are configured correctly. In the output below, interface 1/1/1 and port channel 1 connect to the DNS/NTP server. 1/1/13-1/1/14 are the links to the SFS leaf switches, and 1/1/11-1/1/12 are the VLTi links. VLAN 4094 and port channel 1000 are automatically configured for the VLTi.
VLAN 1911 is the external management VLAN that contains the DNS/NTP server. VLAN 4094 and port channel 1000 are automatically configured for the VLTi.
External-A# show ip interface brief
Interface Name IP-Address OK Method Status Protocol
================================================================================
Ethernet 1/1/1 unassigned YES unset up up
Ethernet 1/1/11 unassigned YES unset up up
Ethernet 1/1/12 unassigned YES unset up up
Ethernet 1/1/13 192.168.1.0/31 YES manual up up
Ethernet 1/1/14 192.168.1.2/31 YES manual up up
Management 1/1/1 100.67.76.41/24 YES manual up up
Vlan 1 unassigned YES unset up up
Vlan 1911 172.19.11.252/24 YES manual up up
Vlan 4000 192.168.3.20/31 YES manual up up
Vlan 4094 unassigned YES unset up up
Port-channel 1 unassigned YES unset up up
Port-channel 1000 unassigned YES unset up up
Loopback 0 10.0.2.1/32 YES manual up up
Theshow ip route command output for the External-A switch appears as shown. No BGP routes from the SFS fabric are learned at this stage of deployment. Interfaces 1/1/13 and 1/1/14 are connected to the SFS leaf switches.
External-A# show ip route
Codes: C - connected
S - static
B - BGP, IN - internal BGP, EX - external BGP
O - OSPF, IA - OSPF inter area, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, E1 - OSPF external type 1,
E2 - OSPF external type 2, * - candidate default,
+ - summary route, > - non-active route
Gateway of last resort is not set
Destination Gateway Dist Last Change
----------------------------------------------------------------------------------
C 10.0.2.1/32 via 10.0.2.1 loopback0 0/0 00:39:19
B IN 10.0.2.2/32 via 192.168.3.21 200/0 00:31:38
C 172.19.11.0/24 via 172.19.11.252 vlan1911 0/0 00:44:00
C 192.168.1.0/31 via 192.168.1.0 ethernet1/1/13 0/0 01:44:44
C 192.168.1.2/31 via 192.168.1.2 ethernet1/1/14 0/0 01:40:50
B IN 192.168.2.0/31 via 192.168.3.21 200/0 00:31:38
B IN 192.168.2.2/31 via 192.168.3.21 200/0 00:31:38
C 192.168.3.20/31 via 192.168.3.20 vlan4000 0/0 00:31:51
Show command output on Leaf1A (BGP example)
Run the show ip bgp summary command to verify that BGP is up for each neighbor. When BGP is up, uptime is shown in the Up/Down
column. The neighbors for Leaf1A shown in the output below are Leaf1B, External-A, and External-B.
Leaf1A# show ip bgp summary
BGP router identifier 172.16.128.0 local AS number 65011
Neighbor AS MsgRcvd MsgSent Up/Down State/Pfx
172.16.0.0 65011 13 16 00:06:59 8
192.168.1.0 65101 12 14 00:07:30 8
192.168.2.0 65101 8 9 00:04:14 8
Run the show ip interface brief command to verify connected interfaces are up, and IP addresses are configured correctly.
In the output below, interfaces 1/1/1-1/1/3 are connected to the VxRail nodes, 1/1/49-1/1/52 are the VLTi, and 1/1/53-1/1/54 are the uplinks to the external switches. SFS uses VLAN 4000-4090, Loopback 1, and Loopback 2 internally. VLAN 4094 and port channel 1000 are automatically configured for the VLTi.
Leaf1A# show ip interface brief
Interface Name IP-Address OK Method Status Protocol
================================================================================
Ethernet 1/1/1 unassigned YES unset up up
Ethernet 1/1/2 unassigned YES unset up up
Ethernet 1/1/3 unassigned YES unset up up
Ethernet 1/1/49 unassigned YES unset up up
Ethernet 1/1/50 unassigned YES unset up up
Ethernet 1/1/51 unassigned YES unset up up
Ethernet 1/1/52 unassigned YES unset up up
Ethernet 1/1/53 192.168.1.1/31 YES manual up up
Ethernet 1/1/54 192.168.2.1/31 YES manual up up
Management 1/1/1 100.67.76.30/24 YES manual up up
Vlan 4000 unassigned YES unset up up
Vlan 4089 unassigned YES unset up up
Vlan 4090 172.16.0.1/31 YES manual up up
Vlan 4094 unassigned YES unset up up
Port-channel 1000 unassigned YES unset up up
Loopback 1 172.16.128.0/32 YES manual up up
Loopback 2 172.30.0.0/32 YES manual up up
Virtual-network 3939 unassigned YES unset up up
Run the show ip route command to verify routes to the external management VLAN, 172.19.11.0/24, have been learned using BGP from the external switches. In this example, two routes to 172.19.11.0/24 are learned, one using each external switch and are underlined in the example below.
Leaf1A# show ip route
Codes: C - connected
S - static
B - BGP, IN - internal BGP, EX - external BGP
O - OSPF, IA - OSPF inter area, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, E1 - OSPF external type 1,
E2 - OSPF external type 2, * - candidate default,
+ - summary route, > - non-active route
Gateway of last resort is not set
Destination Gateway Dist Last Change
----------------------------------------------------------------------------------
B EX 10.0.2.1/32 via 192.168.1.0 20/0 00:43:16
via 192.168.2.0
B EX 10.0.2.2/32 via 192.168.1.0 20/0 00:43:16
via 192.168.2.0
C 172.16.0.0/31 via 172.16.0.1 vlan4090 0/0 02:19:46
C 172.16.128.0/32 via 172.16.128.0 loopback1 0/0 02:20:07
B IN 172.16.128.1/32 via 172.16.0.0 200/0 02:19:44
B EX 172.19.11.0/24 via 192.168.1.0 20/0 00:43:32
via 192.168.2.0
C 172.30.0.0/32 via 172.30.0.0 loopback2 0/0 02:20:07
C 192.168.1.0/31 via 192.168.1.1 ethernet1/1/53 0/0 01:12:49
B IN 192.168.1.2/31 via 172.16.0.0 200/0 01:09:12
C 192.168.2.0/31 via 192.168.2.1 ethernet1/1/54 0/0 01:10:18
B IN 192.168.2.2/31 via 172.16.0.0 200/0 01:07:51
B EX 192.168.3.20/31 via 192.168.1.0 20/0 00:43:21
via 192.168.2.0
Validate static route example
Once the uplink interfaces have been configured on the external switches and in the SFS web UI, connectivity can be verified using the switch CLI.
Show command output on External-A (static route example)
Run the show ip interface brief command to verify connected interfaces are up, and IP addresses are configured correctly. In the output below, interface 1/1/1 and port channel 1 connect to the DNS/NTP server. 1/1/13-1/1/14 are the links to the SFS leaf switches, and 1/1/11-1/1/12 are the VLTi links.
VLAN 1911 is the external management VLAN that contains the DNS/NTP server. VLAN 4094 and port channel 1000 are automatically configured for the VLTi.
External-A# show ip interface brief
Interface Name IP-Address OK Method Status Protocol
================================================================================
Ethernet 1/1/1 unassigned YES unset up up
Ethernet 1/1/11 unassigned YES unset up up
Ethernet 1/1/12 unassigned YES unset up up
Ethernet 1/1/13 192.168.1.0/31 YES manual up up
Ethernet 1/1/14 192.168.1.2/31 YES manual up up
Management 1/1/1 100.67.76.41/24 YES manual up up
Vlan 1 unassigned YES unset up up
Vlan 1911 172.19.11.252/24 YES manual up up
Vlan 4094 unassigned YES unset up up
Port-channel 1 unassigned YES unset up up
Port-channel 1000 unassigned YES unset up up
Run the show ip route command to verify static routes to the external management VLAN, 172.18.11.0/24, are properly configured.
External-A# show ip route
Codes: C - connected
S - static
B - BGP, IN - internal BGP, EX - external BGP
O - OSPF, IA - OSPF inter area, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, E1 - OSPF external type 1,
E2 - OSPF external type 2, * - candidate default,
+ - summary route, > - non-active route
Gateway of last resort is not set
Destination Gateway Dist Last Change
----------------------------------------------------------------------------------
S 172.18.11.0/24 via 192.168.1.1 ethernet1/1/13 1/0 3 days 23:35:18
via 192.168.1.3 ethernet1/1/14
C 172.19.11.0/24 via 172.19.11.252 vlan1911 0/0 3 days 23:26:55
C 192.168.1.0/31 via 192.168.1.0 ethernet1/1/13 0/0 21:58:31
C 192.168.1.2/31 via 192.168.1.2 ethernet1/1/14 0/0 21:58:33
Show command output on Leaf1A (static route example)
Run the show ip interface brief command to verify connected interfaces are up, and IP addresses are configured correctly.
In the output below, interfaces 1/1/1-1/1/3 are connected to the VxRail nodes, 1/1/49-1/1/52 are the VLTi links, and 1/1/53-1/1/54 are the uplinks to the external switches.
Leaf1A# show ip interface brief
Ethernet 1/1/1 unassigned YES unset up up
Ethernet 1/1/2 unassigned YES unset up up
Ethernet 1/1/3 unassigned YES unset up up
Ethernet 1/1/49 unassigned YES unset up up
Ethernet 1/1/50 unassigned YES unset up up
Ethernet 1/1/51 unassigned YES unset up up
Ethernet 1/1/52 unassigned YES unset up up
Ethernet 1/1/53 192.168.1.1/31 YES manual up up
Ethernet 1/1/54 192.168.2.1/31 YES manual up up
Management 1/1/1 100.67.76.30/24 YES manual up up
Vlan 4000 unassigned YES unset up up
Vlan 4090 172.16.0.1/31 YES manual up up
Vlan 4094 unassigned YES unset up up
Port-channel 1000 unassigned YES unset up up
Loopback 1 172.16.128.0/32 YES manual up up
Loopback 2 172.30.0.0/32 YES manual up up
Virtual-network 3939 unassigned YES unset up up
Run the show ip route command to verify static routes to the external management VLAN, 172.19.11.0/24, are correctly configured. The static routes are underlined in the command output below.
Leaf1A# show ip route
Codes: C - connected
S - static
B - BGP, IN - internal BGP, EX - external BGP
O - OSPF, IA - OSPF inter area, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, E1 - OSPF external type 1,
E2 - OSPF external type 2, * - candidate default,
+ - summary route, > - non-active route
Gateway of last resort is not set
Destination Gateway Dist Last Change
----------------------------------------------------------------------------------
C 172.16.0.0/31 via 172.16.0.1 vlan4090 0/0 02:09:34
C 172.16.0.0/31 via 172.16.0.1 vlan4090 0/0 00:40:43
B IN 172.16.128.0/32 via 172.16.0.0 200/0 00:40:42
C 172.16.128.1/32 via 172.16.128.1 loopback1 0/0 00:40:50
S 172.19.11.0/24 via 192.168.1.0 ethernet1/1/53 1/0 00:37:51
via 192.168.2.0 ethernet1/1/54
C 172.30.0.0/32 via 172.30.0.0 loopback2 0/0 00:40:50
C 192.168.1.0/31 via 192.168.1.1 ethernet1/1/53 0/0 00:37:56
B IN 192.168.1.2/31 via 172.16.0.0 200/0 00:35:52
C 192.168.2.0/31 via 192.168.2.1 ethernet1/1/54 0/0 00:36:57
B IN 192.168.2.2/31 via 172.16.0.0 200/0 00:34:51