Selecting the appropriate server and network configuration for generative AI model customization is crucial to ensure adequate resources are allocated for model training. This section provides example configurations for both management and compute workloads and network architecture.

Management server configuration

The following sections describe the management server configuration.

PowerEdge R660 head and control plane node

The following table provides the recommended minimum configuration for the management head node and the control plane node:

Table 3. PowerEdge R660 head node and control plane configuration

Consider the following recommendations for head and control plane node configuration:

• We recommended three PowerEdge servers for management. Install NVIDIA Base Command Manager Essentials on one of the servers. If you require high availability, you can install NVIDIA Base Command Manager Essentials on two nodes as active-passive. Install Kubernetes control plane on all the three nodes. Therefore, one node acts as both an NVIDIA Base Command Manager head node and a Kubernetes control plane node. We do not recommend a single node Kubernetes Control plane.
• We recommend the same hardware configuration for all three head nodes for ease of configuration and maintenance.
• Because both the head node and control plane node do not require heavy computing, a single-processor server is sufficient.
• For the head node, we recommend a storage-rich configuration to facilitate convenient storage of images and other essential tools. We recommend a minimum of four SSD drives. You can choose more drives or upgrade to NVMe for better performance.
• One or two management servers can connect to the InfiniBand fabric. OpenSM is an InfiniBand compliant Subnet Manager service that can be run on any server on the InfiniBand fabric, however we recommend running this service on any of the management servers. 

GPU worker node configuration

PowerEdge XE9680 servers can be configured as worker nodes for Generative AI model customization. The following table provides a recommended configuration for a PowerEdge XE9680 GPU worker node:

Table 4. PowerEdge XE9680 GPU worker node

Internally, each XE9680 has four PCIe switches. Since two GPUs are connected to each PCIe switch, for maximum throughput and performance, each PCIe switch has been subdivided into two virtual switches. Therefore, for optimal GPU network performance, we recommend that each GPU has a dedicated network adapter. We recommend four InfiniBand adapters for the XE8640 and eight InfiniBand adapters for the XE9680.

While LLM tasks primarily rely on GPUs and do not significantly tax the CPU and memory, it is advisable to equip the system with high-performance CPUs and larger memory capacities. This provisioning ensures sufficient headroom for various data processing activities, machine learning operations, monitoring, and logging tasks. Our goal is to guarantee that the servers boast ample CPU and memory resources for these functions, preventing any potential disruptions to the critical AI operations carried out on the GPUs.

Networking design

The following figure shows the network architecture. It shows the network connectivity for compute servers. The figure also shows three PowerEdge head nodes, which incorporate NVIDIA Base Command Manager Essentials and Kubernetes control plane nodes.

Figure 5.   Network design

This validated design requires the following networks to manage the cluster and facilitate communication and coordination between different components and nodes in the cluster:

• Management network—This network is used for communication between the management server and the cluster nodes. It allows the management server to send commands, configurations, and updates to the nodes. It also enables the nodes to report status, resource usage, and other information back to the management server. This network also serves as the Kubernetes network for internode communication in the cluster. It allows the nodes, Kubernetes pods, and services to exchange data, synchronize tasks, and collaborate efficiently during cluster operations.
• External/data center network—The external network connects the cluster to the Internet, allowing the cluster nodes to communicate with external systems, services, and the Internet. This network is essential for accessing external resources, downloading software updates, and interacting with users or applications outside the cluster.
• Storage network—In some configurations, a dedicated storage network might be used to facilitate data transfer between the cluster nodes and storage devices. This network helps to optimize data access and reduce latency for storage operations.
• OOB and PXE network—The out-of-band (OOB) network is a separate and dedicated network infrastructure used for managing and monitoring servers. It is a 1Gb Ethernet network that connects to the Integrated Dell Remote Access Controller (iDRAC) of the PowerEdge servers in the cluster. This network is also used for PXE to automate the provisioning and deployment of operating systems.
• InfiniBand network – This network is used for internode and GPU communication for distributed training.

Rack design 

The following figure shows an example rack design for two PowerEdge XE9680 servers, equivalent to the configuration shown in Figure 5:

Figure 6.   Example rack configuration for Validated Design for Model Customization

This rack was created by using the Dell Enterprise Infrastructure Planning Tool (with the illustrations of the switches enhanced).  You can use the tool to configure your solution and receive weight, power requirement, airflow, and other details.

This example shows two servers in a single rack and requires two 17kW Power Distribution Units (PDUs). However, customers must carefully evaluate their own power and cooling requirements and their preference for rack layout, power distribution, airflow management, and cabling design.

Where significant growth is anticipated in the size of the deployment, customers should consider separate racks for compute, storage, and management nodes to allow sufficient capacity for that growth.

The following table provides the APC Rack and PDU recommendations for the Americas region. We recommend that you consult your Dell or APC representative to understand your unique data center requirements to provide an accurate PDU recommendation.

Table 5. Example Rack and PDU recommendations for the PowerEdge XE9680 server