Network layer

The management network layer of the VxBlock System 1000 includes at least one pair of Cisco Nexus 31108TC-V Switches.

If more than one pair of Cisco Nexus 31108TV-C Switches is required, a pair of Cisco Nexus 3232C or 9336C-FX2 Switches is used. Cisco Nexus 3232C or 9336C-FX2 Switches aggregate management traffic. The Cisco Nexus switches provide 1/10 Gbps connectivity to the management interfaces of the system components, AMP Central, AMP-VX, or AMP-3S.

The following table shows management network layer components:

Cisco Nexus 31108TC-V Switch

The base Cisco Nexus 31108TC-V Switches are deployed in pairs to provide 40/100 GbE connections. Each switch provides 48 100 Mbps/1 GbE/10 GbE Base-T fixed ports and six QSFP28 ports to provide 40/100 GbE connections.

The following components connect to the Cisco Nexus 31108TC-V switches using Cat6 connections at 1/10 Gbps:

• FI management
• Cisco Nexus 9000 Series Switch management
• Cisco MDS SAN switch management
• Storage array management
• IPI Appliance management
• Data Protection appliance connections
• AMP-VX iDRAC
• AMP-VX, AMP-3S, or AMP Central workload
• AMP-VX VMware vSAN
• AMP Central or AMP-VX Data Protection appliance connections
• AMP-3S OOB management iSCSI
• AMP Central or AMP-3S CIMC
• AMP-3S Dell EMC Unity 300 iSCSI
• AMP-3S Dell EMC Unity XT 380 management and iSCSI

The Cisco Nexus 31108TC-V Switches uplink to the external network using one of the following:

• Cat6 at 1/10 Gbps
• Multimode fiber at 10/40/100 Gbps
• AOC direct-attaches at 100 Gbps

At least two uplinks per management switch are required.

Cisco Nexus 3232C Switch

The Cisco Nexus 3232C Switch is used in the management plane to aggregate pairs of Cisco Nexus 31108TC-V Switches. These switches are required only if the solution requires more than 96 management ports.

Each Cisco Nexus 31108TC-V Switch has one or two uplinks to each Cisco Nexus 3232C Switch.

The Cisco Nexus 3232C Switch provides:

• 100 Gbps QSFP ports to aggregate traffic between multiple pairs of Cisco Nexus 31108TC-V Switches
• 100 Gbps QSFP ports for vPC peer link
• 10/40/100 Gbps uplinks to the external network using QSFP or QSA adapter

Cisco Nexus 9336C-FX2 Switch

The Cisco Nexus 9336C-FX2 Switch is a ToR data plane switch that can be configured using Cisco UCS third or fourth-generation components. The Cisco Nexus 9336C-FX2 Switch provides 36 40/100 Gbps QSFP 28 ports. The Cisco Nexus 9336C-FX2 Switch is used for more switch pairs to support Isilon connectivity. The Cisco Nexus 9336C-FX2 Switch is used in the management plane to aggregate pairs of Cisco Nexus 31108TC-V Switches. These switches are required only if the solution requires more than 96 management ports.

Each Cisco Nexus 31108TC-V Switch has one or two uplinks to each Cisco Nexus 9336C-FX2 Switch. The Cisco Nexus 9336C-FX2 Switch provides:

• 100 Gbps QSFP ports to aggregate traffic between multiple pairs of Cisco Nexus 31108TC-V management switches
• 100 Gbps QSFP ports for the vPC peer link
• 10/40/100 Gbps uplinks to the external network using QSFP or MPO hydra

Cisco UCS third-generation uplinks

Cisco UCS 6332-16UP Fabric Interconnect uplinks to the Cisco Nexus 9336C-FX2 Switch use Cisco 40 Gbps QSFP Active Optical cables for the first two domains. Domains three through seven use QSFP optical bi-directional transceivers. Each pair of FIs consumes the following:

• Four QSFP ports on each Cisco Nexus 9336C-FX2 Switch for a standard uplink topology, or
• Six QSFP ports on each Cisco Nexus 9336C-FX2 Switch for enhanced bandwidth uplink topology.

Cisco UCS fourth-generation uplinks

Uplinks from the Cisco UCS 6454 Fabric Interconnects to the Cisco Nexus 9336C-FX2 Switch use:

• Cisco 100 Gbps QSFP Active Optical cables for the first two Cisco UCS domains
• QSFP optical BiDi-transceivers for Cisco UCS domains three through seven

The Cisco UCS 6454 Fabric Interconnects can consume four, eight, or twelve total ports on a pair of Cisco Nexus 9336C-FX2 Switches:

• FI 2-Uplink (each FI has two uplink ports) (System Standard): Each pair of Cisco UCS 6454 FIs consumes four total QSFP ports on a pair Cisco Nexus 9336C-FX2 Switches for a standard uplink topology.
• FI 4-Uplink (each FI has four uplink ports): Each pair of Cisco UCS 6454 Fabric Interconnects consumes eight total QSFP ports on a pair Cisco Nexus 9336C-FX2 Switches for an enhanced bandwidth uplink topology.
• FI 6-Uplink (each FI has six uplink ports): Each pair of Cisco UCS 6454 Fabric Interconnects consumes twelve total QSFP ports on a pair Cisco Nexus 9336C-FX2 Switches for an enhanced bandwidth uplink topology.

Storage array NAS connections

A Cisco QSFP-40G-SR4 transceiver connects the 10 Gbps NAS ports on a VxBlock System 1000 storage array to the Cisco Nexus 9336C-FX2 Switch. A combination of MPO-to-LC hydra cables, MPO trunks, and MPO-to-LC cassettes provide fiber connectivity to the endpoint. The NAS device has 10 Gbps optical SFP+ modules. A direct-attach QSFP cable that is provided with the node connects to a 40 Gbps Isilon node. NAS devices are cross-connected to both switches.

AMP-VX connections

The Cisco QSFP-40G-SR4 transceiver connects each AMP-VX node to each Cisco Nexus 9336C-FX2 Switch. A combination of MPO-to-LC hydra cables, MPO trunks, and MPO-to-LC cassettes provide fiber connectivity to the endpoint.

The configuration provides four 10 Gbps LC fiber connections from each QSFP module. These connections are used for data-plane traffic, such as connections to the VMware vSphere ESXi Management network from workloads residing on the AMP-VX.

AMP-3S connections

The Cisco QSFP-40G-SR4 transceiver connects each AMP-3S node to each Cisco Nexus 9336C-FX2 Switch. A combination of MPO-to-LC hydra cables, MPO trunks, and MPO-to-LC cassettes provide fiber connectivity to the endpoint. The configuration provides four 10 Gbps LC fiber connections from each QSFP module. These connections are used for data-plane traffic and in-band management for AMP-3S and VxBlock 1000. For example, the connections link the VMware vSphere ESXi Management network from workloads residing on the AMP-3S.

AMP Central connections

The Cisco QSFP-40G-SR4-S Transceiver connects each AMP Central node to each ToR Cisco Nexus 9336C-FX2 Switch. A combination of MPO-to-LC hydra cables, MPO trunks, and MPO-to-LC cassettes provide fiber connectivity to the endpoint. This provides four 25 Gbps LC fiber connections from each QSFP module. These connections are used for data plane traffic and in-band management for AMP Central and VxBlock 1000. Connections to the VMware vSphere ESXi Management network from workloads residing on AMP Central also use data plane traffic and in-band management.

vPC peer link

Two 100 Gbps copper direct-attach connections form the vPC peer link the Cisco Nexus 93180LC-EX Switches.

Cisco UCS third and fourth-generation external uplinks

The following table shows connections to the external network for Cisco UCS Third-Generation uplinks:

The following table shows connections to the external network for Cisco UCS Fourth-Generation uplinks:

Cisco UCS fourth-generation Integrated Data Protection connections

Most Integrated Data Protection solutions include a pair of dedicated Cisco Nexus 93180YC-EX Switches, which are uplinked to the Cisco Nexus 9336C-FX2 Switches using 40 Gbps QSFP direct-attach cables.

For small deployments, direct connectivity from the Integrated Data Protection components to the switches is made with Cisco QSFP-40G-SR4 transceiver using the same connection method as the NAS connections. This provides four 10 Gbps LC fiber connections from each QSFP module. In-band management connections from the Integrated Data Protection components use 1 Gbps GLC-T SFP+ to Cat6 connections.

Cisco UCS third-generation Integrated Data Protection connections

Most Integrated Data Protection solutions include a pair of dedicated Cisco Nexus 93180YC-EX Switches that are uplinked to the Cisco Nexus 9336C-FX2 Switches using 40 Gbps QSFP direct-attach cables.

For small deployments, direct connectivity from Integrated Data Protection components to the switches is made with Cisco QSFP-40G-SR4 transceiver using the same connection method as the NAS connections. This method provides four 10 Gbps LC fiber connections from each QSFP module. In-band management connections from the Integrated Data Protection components use 1 Gbps GLC-T SFP+ to Cat6 connections.

Isilon connections

For small Isilon deployments, the Cisco Nexus 9336C-FX2 Switches can connect:

• Isilon nodes at 40 Gbps using the QSFP-40G-SR4 transceiver and MPO cabling
• Isilon nodes at 10 Gbps using the connection method described for NAS connections

Larger Isilon deployments require more Cisco Nexus 9336C-FX2 Switch pairs.

Software licensing

The Cisco Nexus 9336C-FX2 Switch introduces a new tiered licensing model. If you choose the Layer 3 uplink model for the VxBlock 1000, you must purchase a license bundle that includes these services for the ToR switch pair. The license packages that are offered with the VxBlock 1000 are NX-OS Essentials and NX-OS Advanced. Both of these license packages include all Layer 3 protocols, and Cisco DCNM LAN.

If you choose the Layer 2 uplink model, you do not must purchase a license bundle. However, you must purchase a Cisco DCNM LAN license for each switch in the solution.

Subordinate switches connected to the ToR Cisco Nexus 9336C-FX2 Switch pair do not require a tiered license package, but do require a Cisco DCNM LAN license.

VxBlock System 1000 with no management aggregation layer

A pair of Cisco Nexus 31108TC-V Switches in Layer 3 mode, which is connected directly to the external network provides a simple network deployment for the VxBlock System 1000.

For a single VxBlock System 1000 with no management aggregation layer, the following figure shows the base management network topology:

Management network SVIs and HSRP addresses reside on the VxBlock System 1000 management switches (Layer 3) or on the external network (Layer 2).

Single VxBlock System 1000 with a management aggregation layer

A single VxBlock System 1000 with a management aggregation layer introduces the management network to enable a larger scale for the management network.

The following figure shows the management network topology for a single VxBlock System 1000 with the management aggregation layer:

Management network Layer 3 SVIs, and HSRP addresses can reside on the VxBlock System 1000 management aggregation switches or on the external network. There is one uplink from each Cisco Nexus 31108TC-V Switch to each Cisco Nexus management aggregation switch at 100 Gbps.

Multiple VxBlock Systems 1000 with a management aggregation layer

This option for multiple VxBlock Systems 1000 puts the management network on the management aggregation switches.

The management aggregation layer contains the following:

• All management network Layer 3 SVIs and HSRP addresses reside on the VxBlock System 1000 management aggregation switches.
• Each connected VxBlock System 1000 has a unique VLAN ID and subnet for OOB management

When adding a second VxBlock System to an existing AMP Central or  AMP-VX deployment, you may need two Cisco Nexus 9336C-FX2 or Cisco Nexus 3232C management aggregation switches to the second VxBlock System.

During deployment, move all existing SVIs and uplinks for the VxBlock System management switches the Cisco Nexus 9336C-FX2 or Cisco Nexus Cisco Nexus 3232C switch pair.

The following figure depicts AMP-VX with aggregation management:

The following diagram shows AMP Central with management aggregation:

• All management network Layer 3 SVIs and HSRP addresses reside on the VxBlock System 1000 management aggregation switches.
• Each connected VxBlock System 1000 has a unique VLAN ID and subnet for out-of-band management

Software licenses

In deployments where the Cisco Nexus 31108TC-V Switches are the only management switches, if you use Layer 3 uplink configuration, purchase a LAN Enterprise Services license.

Layer 3 protocols require a LAN Enterprise services license. If you choose the Layer 2 uplink model, you do not need to purchase a LAN Enterprise Service license.

[2]All Cisco Nexus 31108TC-V Switches must include a Cisco DCNM LAN license.

For solutions including a management aggregation switch pair, see Software Licenses in the Cisco Nexus 9336C-FX2 data plane switch.