Selecting the appropriate server and network configuration for generative AI inference is crucial to ensure adequate resources are allocated for both management and inference tasks. This section provides example configurations for both management and compute workloads and network architecture.

Management server configuration

PowerEdge R660 head and control plane node

The following table provides the recommended minimum configuration for the management head node and Kubernetes control plan node.

Table 4. PowerEdge R660 head node and Kubernetes control plane configuration

Consider the following recommendations:

• For the number of nodes, we recommend three PowerEdge servers for management. NVIDIA Base Command Manager Essentials is installed on one of the servers. If the customer requires high availability, NVIDIA Base Command Manager Essentials can be installed on two nodes as an active-passive cluster. Kubernetes control plane is installed on all three nodes. One node acts as both the NVIDIA Base Command Manager head node and the Kubernetes control plane node. We do not recommend a single-node Kubernetes control plane.
• We recommend the same hardware configuration for all three head nodes for ease of configuration and maintenance.
• Because both the head node and control plane node do not require heavy computing, a single-processor server is sufficient.
• For the head node, we recommend opting for a storage-rich configuration, as this configuration will facilitate convenient storage of images and other essential tools. We recommend four SSD drives. Customers can choose more drives or upgrade to NVMe for better performance.

GPU worker node configuration

Dell Technologies provides a selection of three GPU-optimized servers suitable for configuration as worker nodes for Generative AI inference: the PowerEdge R760xa, PowerEdge XE9680, and PowerEdge XE8640 servers. Customers have the flexibility to choose one of these PowerEdge servers based on the specific model size that they require. Larger models, characterized by a greater parameter size, require servers equipped with a higher GPU count and enhanced connectivity. For specific examples of LLM models that can be deployed on each server model, see Table 3.

The GPU-optimized servers act as worker nodes in a Kubernetes cluster. The number of servers depends on the number of models and the number of concurrent requests served by those models. We have validated an eight-GPU worker node cluster. The minimum number of worker nodes in the cluster is one.

PowerEdge R760xa GPU worker node

The following table shows a recommended configuration for a PowerEdge R760xa GPU worker node.

Table 5. PowerEdge R760xa GPU worker node

PowerEdge XE8640 GPU worker node

The following table shows a recommended configuration for a PowerEdge XE8640 GPU worker node.

Table 6. PowerEdge XE8640 GPU worker node

PowerEdge XE9680 GPU worker node

The following table shows a recommended configuration for a PowerEdge XE9680 GPU worker node.

Table 7. PowerEdge XE9680 GPU worker node

The CPU memory allocation in the XE9680 GPU worker node configuration exceeds that of the XE8640 configuration. This increase is attributed to the presence of twice as many GPUs, which implies a heightened demand for overall inferencing capacity and, therefore, greater CPU memory requirements.

While inferencing tasks primarily rely on GPUs and do not significantly tax the CPU and memory, it is advisable to equip the system with high-performance CPUs and larger memory capacities. This provisioning ensures sufficient headroom for various data processing activities, machine learning operations, monitoring, and logging tasks. Our goal is to guarantee that the servers boast ample CPU and memory resources for these functions, preventing any potential disruptions to the critical inferencing operations carried out on the GPUs.

Networking design

The following figure shows the network architecture. It shows the network connectivity for compute servers. The figure also shows three PowerEdge head nodes, which incorporate NVIDIA Base Command Manager Essentials and Kubernetes control plane nodes.

Figure 3. Network architecture

This validated design requires the following networks to manage the cluster and facilitate communication and coordination between different components and nodes within the cluster:

• Management network—This network is used for communication between the management server and the cluster nodes. It allows the management server to send commands, configurations, and updates to the nodes. It also enables the nodes to report status, resource usage, and other information back to the management server. This network also serves as the Kubernetes network for internode communication in the cluster. It allows the nodes, Kubernetes pods, and services to exchange data, synchronize tasks, and collaborate efficiently during cluster operations.
• External/data center network—The external network connects the cluster to the Internet, allowing the cluster nodes to communicate with external systems, services, and the Internet. This network is essential for accessing external resources, downloading software updates, and interacting with users or applications outside the cluster.
• Storage network—In some configurations, a dedicated storage network might be used to facilitate data transfer between the cluster nodes and storage devices. This network helps to optimize data access and reduce latency for storage operations.
• OOB and PXE network—The out-of-band (OOB) network is a separate and dedicated network infrastructure used for managing and monitoring servers. It is a 1Gb Ethernet network that connects to the Integrated Dell Remote Access Controller (iDRAC) of the PowerEdge servers in the cluster. This network is also used for PXE to automate the provisioning and deployment of operating systems.