The management network layer of the VxBlock System 1000 includes at least one pair of Cisco Nexus 31108TC-V Switches.
If more than one pair of Cisco Nexus 31108TV-C Switches is required, a pair of Cisco Nexus 3232C or 9336C-FX2 Switches is used. Cisco Nexus 3232C or 9336C-FX2 Switches aggregate management traffic. The Cisco Nexus switches provide 1/10 Gbps connectivity to the management interfaces of the system components, AMP Central, AMP-VX, or AMP-3S.
The following table shows management network layer components:
Switch | Description |
Cisco Nexus 31108TC-V | |
Cisco Nexus 3232C | |
Cisco Nexus 9336C-FX2 |
The base Cisco Nexus 31108TC-V Switches are deployed in pairs to provide 40/100 GbE connections. Each switch provides 48 100 Mbps/1 GbE/10 GbE Base-T fixed ports and six QSFP28 ports to provide 40/100 GbE connections.
The following components connect to the Cisco Nexus 31108TC-V switches using Cat6 connections at 1/10 Gbps:
The Cisco Nexus 31108TC-V Switches uplink to the external network using one of the following:
At least two uplinks per management switch are required.
The Cisco Nexus 3232C Switch is used in the management plane to aggregate pairs of Cisco Nexus 31108TC-V Switches. These switches are required only if the solution requires more than 96 management ports.
Each Cisco Nexus 31108TC-V Switch has one or two uplinks to each Cisco Nexus 3232C Switch.
The Cisco Nexus 3232C Switch provides:
The Cisco Nexus 9336C-FX2 Switch is a ToR data plane switch that can be configured using Cisco UCS third or fourth-generation components. The Cisco Nexus 9336C-FX2 Switch provides 36 40/100 Gbps QSFP 28 ports. The Cisco Nexus 9336C-FX2 Switch is used for more switch pairs to support Isilon connectivity. The Cisco Nexus 9336C-FX2 Switch is used in the management plane to aggregate pairs of Cisco Nexus 31108TC-V Switches. These switches are required only if the solution requires more than 96 management ports.
Each Cisco Nexus 31108TC-V Switch has one or two uplinks to each Cisco Nexus 9336C-FX2 Switch. The Cisco Nexus 9336C-FX2 Switch provides:
Cisco UCS third-generation uplinks
Cisco UCS 6332-16UP Fabric Interconnect uplinks to the Cisco Nexus 9336C-FX2 Switch use Cisco 40 Gbps QSFP Active Optical cables for the first two domains. Domains three through seven use QSFP optical bi-directional transceivers. Each pair of FIs consumes the following:
Cisco UCS fourth-generation uplinks
Uplinks from the Cisco UCS 6454 Fabric Interconnects to the Cisco Nexus 9336C-FX2 Switch use:
The Cisco UCS 6454 Fabric Interconnects can consume four, eight, or twelve total ports on a pair of Cisco Nexus 9336C-FX2 Switches:
Storage array NAS connections
A Cisco QSFP-40G-SR4 transceiver connects the 10 Gbps NAS ports on a VxBlock System 1000 storage array to the Cisco Nexus 9336C-FX2 Switch. A combination of MPO-to-LC hydra cables, MPO trunks, and MPO-to-LC cassettes provide fiber connectivity to the endpoint. The NAS device has 10 Gbps optical SFP+ modules. A direct-attach QSFP cable that is provided with the node connects to a 40 Gbps Isilon node. NAS devices are cross-connected to both switches.
AMP-VX connections
The Cisco QSFP-40G-SR4 transceiver connects each AMP-VX node to each Cisco Nexus 9336C-FX2 Switch. A combination of MPO-to-LC hydra cables, MPO trunks, and MPO-to-LC cassettes provide fiber connectivity to the endpoint.
The configuration provides four 10 Gbps LC fiber connections from each QSFP module. These connections are used for data-plane traffic, such as connections to the VMware vSphere ESXi Management network from workloads residing on the AMP-VX.
AMP-3S connections
The Cisco QSFP-40G-SR4 transceiver connects each AMP-3S node to each Cisco Nexus 9336C-FX2 Switch. A combination of MPO-to-LC hydra cables, MPO trunks, and MPO-to-LC cassettes provide fiber connectivity to the endpoint. The configuration provides four 10 Gbps LC fiber connections from each QSFP module. These connections are used for data-plane traffic and in-band management for AMP-3S and VxBlock 1000. For example, the connections link the VMware vSphere ESXi Management network from workloads residing on the AMP-3S.
AMP Central connections
The Cisco QSFP-40G-SR4-S Transceiver connects each AMP Central node to each ToR Cisco Nexus 9336C-FX2 Switch. A combination of MPO-to-LC hydra cables, MPO trunks, and MPO-to-LC cassettes provide fiber connectivity to the endpoint. This provides four 25 Gbps LC fiber connections from each QSFP module. These connections are used for data plane traffic and in-band management for AMP Central and VxBlock 1000. Connections to the VMware vSphere ESXi Management network from workloads residing on AMP Central also use data plane traffic and in-band management.
vPC peer link
Two 100 Gbps copper direct-attach connections form the vPC peer link the Cisco Nexus 93180LC-EX Switches.
Cisco UCS third and fourth-generation external uplinks
The following table shows connections to the external network for Cisco UCS Third-Generation uplinks:
Bandwidth | Transceiver | Media | Minimum uplinks per switch pair | Maximum uplinks per switch pair | Maximum distance |
100 Gbps | QSFP-100G-SR4-S | OM4 MPO MMF | 4 | 16 | 100 M |
100 Gbps | QSFP-40/100G-SRBD | LMMF | 4 | 16 | 100 M |
40 Gbps | QSFP-40G-SRBD | MMF | 4 | 16 | 100 M for OM3, 150 M for OM4 |
4 x 10 Gbps | QSFP-40G-SR4 | MPO MMF terminated with LC | 8 SFP+ | 32 SFP + | 100 M for OM3, 150 M for OM4 |
40 Gbps | QSFP-40G-SR4 | MPO MMF | 4 | 16 | 100 M |
4 x 10 Gbps | QSFP-40G-SR4 | MPO MMF terminated with LC | 8 SFP+ | 32 SFP + | 100 M for OM3, 150 M for OM4 |
The following table shows connections to the external network for Cisco UCS Fourth-Generation uplinks:
Bandwidth | Transceiver | Media | Minimum uplinks per switch pair | Maximum uplinks per switch pair | Maximum distance |
100 Gbps | QSFP-100G-SR4-S | OM4 MPO MMF | 4 | 16 | 100 M |
100 Gbps | QSFP-40/100G-SRBD | LMMF | 4 | 16 | 100 M |
40 Gbps | QSFP-40G-SRBD | MMF | 4 | 16 | 100 M for OM3, 150 M for OM4 |
40 Gbps | QSFP-40G-SR4 | MPO MMF | 4 | 16 | 100 M |
10 Gbps | QSFP-10G-SR4 | MPO MMF terminated with LC | 4 QSFP | 8 QSFP | 100 M for OM3, 150 M for OM4 |
Cisco UCS fourth-generation Integrated Data Protection connections
Most Integrated Data Protection solutions include a pair of dedicated Cisco Nexus 93180YC-EX Switches, which are uplinked to the Cisco Nexus 9336C-FX2 Switches using 40 Gbps QSFP direct-attach cables.
For small deployments, direct connectivity from the Integrated Data Protection components to the switches is made with Cisco QSFP-40G-SR4 transceiver using the same connection method as the NAS connections. This provides four 10 Gbps LC fiber connections from each QSFP module. In-band management connections from the Integrated Data Protection components use 1 Gbps GLC-T SFP+ to Cat6 connections.
Cisco UCS third-generation Integrated Data Protection connections
Most Integrated Data Protection solutions include a pair of dedicated Cisco Nexus 93180YC-EX Switches that are uplinked to the Cisco Nexus 9336C-FX2 Switches using 40 Gbps QSFP direct-attach cables.
For small deployments, direct connectivity from Integrated Data Protection components to the switches is made with Cisco QSFP-40G-SR4 transceiver using the same connection method as the NAS connections. This method provides four 10 Gbps LC fiber connections from each QSFP module. In-band management connections from the Integrated Data Protection components use 1 Gbps GLC-T SFP+ to Cat6 connections.
Isilon connections
For small Isilon deployments, the Cisco Nexus 9336C-FX2 Switches can connect:
Larger Isilon deployments require more Cisco Nexus 9336C-FX2 Switch pairs.
Software licensing
The Cisco Nexus 9336C-FX2 Switch introduces a new tiered licensing model. If you choose the Layer 3 uplink model for the VxBlock 1000, you must purchase a license bundle that includes these services for the ToR switch pair. The license packages that are offered with the VxBlock 1000 are NX-OS Essentials and NX-OS Advanced. Both of these license packages include all Layer 3 protocols, and Cisco DCNM LAN.
If you choose the Layer 2 uplink model, you do not must purchase a license bundle. However, you must purchase a Cisco DCNM LAN license for each switch in the solution.
Subordinate switches connected to the ToR Cisco Nexus 9336C-FX2 Switch pair do not require a tiered license package, but do require a Cisco DCNM LAN license.
A pair of Cisco Nexus 31108TC-V Switches in Layer 3 mode, which is connected directly to the external network provides a simple network deployment for the VxBlock System 1000.
For a single VxBlock System 1000 with no management aggregation layer, the following figure shows the base management network topology:
Management network SVIs and HSRP addresses reside on the VxBlock System 1000 management switches (Layer 3) or on the external network (Layer 2).
A single VxBlock System 1000 with a management aggregation layer introduces the management network to enable a larger scale for the management network.
The following figure shows the management network topology for a single VxBlock System 1000 with the management aggregation layer:
Management network Layer 3 SVIs, and HSRP addresses can reside on the VxBlock System 1000 management aggregation switches or on the external network. There is one uplink from each Cisco Nexus 31108TC-V Switch to each Cisco Nexus management aggregation switch at 100 Gbps.
This option for multiple VxBlock Systems 1000 puts the management network on the management aggregation switches.
The management aggregation layer contains the following:
When adding a second VxBlock System to an existing AMP Central or AMP-VX deployment, you may need two Cisco Nexus 9336C-FX2 or Cisco Nexus 3232C management aggregation switches to the second VxBlock System.
During deployment, move all existing SVIs and uplinks for the VxBlock System management switches the Cisco Nexus 9336C-FX2 or Cisco Nexus Cisco Nexus 3232C switch pair.
The following figure depicts AMP-VX with aggregation management:
The following diagram shows AMP Central with management aggregation:
In deployments where the Cisco Nexus 31108TC-V Switches are the only management switches, if you use Layer 3 uplink configuration, purchase a LAN Enterprise Services license.
Layer 3 protocols require a LAN Enterprise services license. If you choose the Layer 2 uplink model, you do not need to purchase a LAN Enterprise Service license.
[2]All Cisco Nexus 31108TC-V Switches must include a Cisco DCNM LAN license.
For solutions including a management aggregation switch pair, see Software Licenses in the Cisco Nexus 9336C-FX2 data plane switch.