Create and configure distributed virtual switches for the compute cluster.
The following VLANs are used in this POC for the compute cluster:
- vlan 848 – flex-node-mgmt
- vlan 858 – flex-stor-Mgmt
- vlan 106 – flex- vMotion
- vlan 152 – flex Data1
- vlan 160 – flex Data2
- vlan 168 – flex Data3
- vlan 176 – flex Data4
The following table shows how the distributed switches are configured in the compute cluster:
Table 24. Distributed switches configuration
- active/active
- Route based on IPHASH
- MLAG
- MTU: 9000
- MLAG with vmnic0/vmnic1
- Distributed Port groups (Ephemeral):
- flex-node-mgmt
- hv-vmotion-106
| - active/active
- Route based on IPHASH
- MLAG with vmnic2/vmnic3
- Distributed Port group (Ephemeral):
- flex-data1-152
- flex-data2-160
- flex-data3-168
- flex-data4-176
|
Create and configure distributed switches
These steps do the following:
- Create two compute distributed switches (DSwitch0 and DSwitch1)
- Configure jumbo frames and the discovery protocol on both compute distributed switches
- Configure LACP on both compute distributed switches
Create and configure DSwitch0 and DSwitch1 for the compute cluster:
- Log in to VMware vSphere Client (HTML5).
- From Home, click Networking and expand the data center.
- Right-click the data center and click Distributed Switch > New Distributed Switch to open the wizard.
- In the New Distributed Switch wizard, type the data values as requested, accept the default settings, and use the following configuration information:
- Name and location—Change Name to DSwitch0-CO.
- Select version—Click NEXT.
- Configure settings—Keep the default values.
- Ready to complete—Click NEXT.
- Verify that the switch is created in the left panel.
- Configure jumbo frames and the discovery protocol on Dswitch0:
- Right-click Dswitch0-CO > Settings > Edit Settings to open the wizard.
- In the DSwitch0 wizard, type the data values as requested, accept the default settings, and use the following configuration information:
- General—Change Network I/O Control to Disabled and keep all other default values.
- Advanced—Change MTU to 9000, type to Link Layer Discovery Protocol, Operation to Both, and then click OK.
- Configure LAG (LACP) on DSwitch0-CO:
- Click DSwitch0-CO > Configure > LACP.
- Click +NEW to open the wizard.
- In the New Distributed Switch wizard, type the data values as requested, accept the default settings, use the following configuration information, and then click OK:
- Change Name to lag-management.
- Leave Number of ports as default (2).
- Leave Mode as default (Active).
- Change Load balancing mode to Source and destination IP address, TCP/UDP ports.
- Repeat steps 1 through 7 to create an additional Distributed Switch named DSwitch1-Data.
Create distributed port groups for DVSwitch0
Create the co-flex-node-mgmt and hv-vmotion-106 distributed port groups on DSwitch0:
- Log in to VMware vSphere Client (HTML5).
- Right-click DVSwitch0 and click Distributed Port Group > New Distribution Port Group to open the wizard.
- In the New Distributed Switch wizard, type the data values as requested, accept the default settings, and use the following configuration information:
- Name and location—Change Name to flex-node-mgmt and click Next.
- Configure settings—Change Port binding to Static, VLAN type to VLAN, VLAN ID to 848, and then click FINISH.
- At the Ready to complete step, review the summary and then click FINISH.
- Right-click the new port group and click Edit Settings to open the wizard.
- In the <port group> wizard, type the data values as requested, accept the default settings, but do the following for the Teaming and Failover field:
- Change Load balancing to Route based on IP hash.
- Change the Failover order to be:
- Active uplinks: lag-managment (LACP).
- Unused uplinks: Uplink 1 and Uplink 2.
- Click OK.
- Repeat steps 2 through 6 for hv-vmotion-106
Create distributed port groups for DVSwitch1
Create the flex-data1, flex-data2, flex-data3, flex-data4, and flex-vmotion distributed port groups on DSwitch1:
- Log in to VMware vSphere Client (HTML5).
- Right-click DVSwitch1 and click Distributed Port Group > New Distribution Port Group&nlacp
bsp;to open the wizard.
- In the New Distributed Switch wizard, type the data values as requested, accept the default settings, and use the following configuration information:
- Name and location—Change Name to co-data1-152 and click Next.
- Configure settings—Change Port binding to Static, VLAN type to VLAN, VLAN ID to 152, and then click NEXT.
- At the Ready to complete step, review the summary and then click FINISH.
- Right-click the new port group and select Edit Settings to open the wizard.
- In the wizard, type the data values as requested, accept the default settings, but do the following:
- For General, change Load balancing to Route based on IP hash.
- For Teaming and Failover:
- Change Load balancing to Route based on IP hash.
- Change the Failover order to be:
- Active uplinks: Mgmt (LACP).
- Unused uplinks: Uplink 1 and Uplink 2.
- Click OK.
- Repeat steps 1 through 4 for co-data2, co-data3, and co-data4.
Migrate compute ESXi hosts
Migrate the physical NICs and ESXi management (vmk0) adapter from vSwitch0 to DSwitch0-CO. These steps are for migrating the ESXi compute sleds from vSwitch0 to DSwitch0-CO. They include reconfiguring LACP one side (NIC) at a time during migration to the distributed switch.
Note: We recommend that you perform these steps on one ESXi host at time.
Only vSwitch0 is created with the flex-node-mgmt adapter configured on each compute ESXi host. This section provides steps to migrate both the physical NICs and ESXi management adapters to DSwitch0. All MX9116n FSE switch ports for the compute ESXi hosts are configured as a trunk access port during initial configuration. These ports are configured later with LACP enabled. The following are the high-level steps:
- Validate the switch port.
- Configure LACP on the 9116-A2 switch port (vmnic1).
- Add vmnic1 to DSwitch0.
- Migrate vmk0 (ESXi mgmt) to DSwitch0.
- Remove vmnic0 from vSwitch0.
- Configure LACP on the 9116-A1 switch port (vmnic0).
- Add vmnic0 to DSwitch0.
Switch configuration example
The following provides a configuration example of the eventual switch configuration:
Note: Do not apply these changes at this time. The following is an example of the switch configuration after all nodes have been converted into LACP.
Table 25. Switch configuration example
interface ethernet1/1/1 description "M740-236 - vmnic0" no shutdown channel-group 11 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/1/3 description "M740-237 - vmnic0" no shutdown channel-group 12 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/71/1 description "M740-238 - vmnic0" no shutdown channel-group 13 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/71/2 description "M740-239 - vmnic0" no shutdown channel-group 14 mode active fec off no switchport mtu 9216 flowcontrol receive off | interface ethernet1/1/1 description "M740-238 - vmnic1" no shutdown channel-group 13 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/1/3 description "M740-239 - vmnic1" no shutdown channel-group 14 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/71/1 description "M740-236 - vmnic1" no shutdown channel-group 11 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/71/2 description "M740-237 - vmnic1" no shutdown channel-group 12 mode active fec off no switchport mtu 9216 flowcontrol receive off |
Physical NIC migration
Migrate both the physical NICs and ESXi management adapters to DSwitch0:
- Log in to the VMware vSphere Client (HTML5).
- Examine the vCenter and physical switches to ensure that all ports across all hosts are up.
- Record to which physical switch port vmnic0 (9116-A1) and vmnic1 (9116-A2) are connected.
- Click Home, then select Hosts and Clusters and expand the compute cluster.
- Select the first compute ESXi host in the left pane and then select the Configure tab in the right pane.
- Select Virtual switches under Networking.
- Expand vSwitch0.
- Under Physical Adapters, expand Uplink 1 (vmnic0), click the ellipsis (…), and select View Settings.
- Click the LLDP tab.
- Record the Port ID (Switch Port) and System Name (Switch) values.
- Repeat steps 3 through 10 for uplink 2 (vmnic1) and then go to step 12.
- Remove vmnic1 from vSwitch0 on the compute ESXi host (sled) as follows:
- Click Home, click Hosts and Clusters, and then expand the compute cluster.
- Select the first compute ESXi host in the left pane and then select the Configure tab in the right pane.
- Select Virtual switches under Networking.
- Expand vSwitch0 and select Manage Physical Adapters to open the wizard.
- In the Manage Physical Network Adapters wizard, type the data values as requested, accept the default settings, and do the following:
- Select vmnic1 and click the red X to remove the NIC.
- Verify that vmnic1 is removed from list.
- Click OK.
- Configure LACP on the 9116-A2 switch port on a compute ESXi sled:
- Use SSH to connect to the 9116-A2 switch.
- Type the following switch commands to configure LACP for the ESXi host in chassis 1/slot 1:
Note: See the port map to obtain port connectivity for each compute ESXi host. Ensure that you use the correct channel-group number for each ESXi host being configured.
# Config t
# interface ethernet1/1/1
# channel-group 12 mode active
- Add vmnic1 to DSwitch0-CO on a compute ESXi host (sled):
- Click Home > Hosts and Clusters.
- Select the first compute ESXi host in left pane and then select the Configure tab in right pane.
- Select Virtual switches under Networking.
- Expand DSwitch0 and select Manage Physical Adapters to open the wizard.
- In the Manage Physical Network Adapters wizard, type the data values as requested, accept the default settings, but do the following:
- Select Mgmt-1 (Mgmt).
- Click + to open list of NICs.
- Select vmnic1.
- Click OK.
- Migrate flex-node-mgmt (vmk0) to DSwitch0 on a compute ESXi host (sled):
- From Home, click Networking and expand the data center.
- Right-click DSwitch0 and select Add and Manage Hosts to open the wizard.
- In the Add and Manage Hosts wizard, type the data values as requested, accept the default settings, and use the following configuration information:
Table 26. Add and Manage Hosts wizard values
Select task | Select Add hosts. Click NEXT. |
Select hosts | Click +New hosts. Select the compute ESXi host and click OK. Click NEXT. |
Manage physical adapters | Select vmnic1. Click Assign uplink. Select Mgmt-1 and click OK. Click NEXT. |
Manage VMkernel adapters | Select vmk0 (Management Network) to highlight. Click Assign port group. Select flex-node-mgmt and click OK. Click NEXT. |
Migrate VM networking | Leave the existing values as default. Click NEXT. |
Ready to complete | Review the summary and click FINISH. |
- Wait 60 seconds and then select the VMkernel adapter link under Networking to ensure that the flex-node-mgmt (vmk0) adapter is on DSwitch0.
- Remove vmnic0 from vSwitch0 on the compute ESXi host (sled):
- Click Home, then click Hosts and Clusters.
- Select the first compute ESXi host in the left pane and then select the Configure tab in the right pane.
- Select Virtual switches under Networking.
- Expand vSwitch0 and select Manage Physical Adapters to open the wizard.
- In the Manage Physical Network Adapters wizard, type the data values as requested, accept the default settings, and do the following:
- Select vmnic0 and click the red X to remove the NIC.
- Verify that vmnic1 is removed from list.
- Click OK.
- Configure LACP on the 9116-A1 switch port on a compute ESXi sled:
- Use SSH to connect to the 9116-A1 switch.
- Type the following example switch commands to configure LACP for ESXi host in chassis 1/slot 1:
Note: See the port map to obtain port connectivity for each compute ESXi host. Ensure that you use the correct channel-group number for each ESXi host being configured.
# Config t
# interface ethernet1/1/1
- # channel-group 12 mode active
- Add vmnic0 to DSwitch0 on a compute ESXi host (sled):
- Click Home > Hosts and Clusters.
- Select the first compute ESXi host in left pane and then select the Configure tab in right pane.
- Select Virtual switches under Networking.
- Expand DSwitch0 and select Manage Physical Adapters to open the wizard.
- In the Manage Physical Network Adapters wizard, type the data values as requested, accept the default settings, and do the following:
- Select Mgmt-0 (Mgmt).
- Click + to open list of NICs.
- Click vmnic0.
- Click OK.
- Repeat the preceding steps for each compute ESXi host.
Creating the VMkernel adapters on distributed switches
Create the network adapters on the following distributed switches:
- DSwitch0: co-mgmt-848, hv-vmotion-106
- DSwitch1: fos-data1-152, fos-data2-160, flex-data3-168, flex-data4-176
Before you begin, verify that the compute ESXi hosts are configured on the distributed switches.
- From Home, click Hosts and Clusters and expand the data center and then expand the compute cluster.
- Select the first compute ESXi host in the left pane and then select the Configure tab in the right pane.
- Click VMkernel adapters under Networking.
- Click Add Networking to open the wizard.
- In the Add Networking wizard, type the data values as requested, accept the default settings, and use the following configuration information
Table 27. Add Networking wizard values
Select connection type | Select VMkernel Network Adapter. Click NEXT. |
Selection target device | Select an existing network. Click Browse. Select hv-vmotion-106 (DVswitch0) and click OK. Click NEXT. |
Port properties | Click the vMotion checkbox. Click NEXT. |
IPv4 settings | Select Use static IPv4 settings: - IPv4 address: 192.168.106.136
- Subnet Mask: 255.255.255.0
Accept the defaults for the Default gateway and DNS Server addresses. Click NEXT. |
Ready to complete | Review the summary and click FINISH. |
- Wait 60 seconds and then verify that co-mgmt-848 and hv-vmotion-106 (vmk1) network adapter is created on DSwitch0.
- Repeat steps 4 through 6 to create flex-data1-152 and flex-data2-160, flex-data3-168 and flex-data4-176 on DSwitch1. Each port group will have a different address than what is in preceding table.
- Repeat steps1 through 7 for each compute ESXi host.