The following figure shows an example of a three-node cluster that is configured with a dual-link full mesh storage network interconnect. External network traffic uses the LOM/rNDC/OCP integrated network adapter, while storage network traffic uses two RDMA-capable PCIe adapters for each server node. In this case, each clustered node has redundant storage network connectivity to every other node in the cluster.
The following figure shows the six storage networks needed to create a dual-link full mesh interconnect between three clustered nodes. The numbered blue lines indicate the recommended cabling order, beginning with the RDMA adapter in the lowest slot number (Port 1). After cabling storage, connect all nodes, LOM/rNDC/OCP, and Ports 1 and 2 to a management/VM network.
Node | First port link connection | Second port link connection | Third port link connection | Fourth port link connection |
Node 1 | Storage1-2 | Storage1-3 | Storage2-1 | Storage3-1 |
Node 2 | Storage1-2 | Storage2-3 | Storage2-1 | Storage3-2 |
Node 3 | Storage1-3 | Storage2-3 | Storage3-1 | Storage3-2 |