Dell Technologies provides a diverse selection of acceleration-optimized servers with an extensive portfolio of accelerators featuring NVIDIA GPUs. In this design, we showcase three Dell PowerEdge servers specifically tailored for generative AI purposes:

• PowerEdge R760xa server, capable of supporting up to four NVIDIA H100 GPUs or four NVIDIA L40 GPUs
• PowerEdge XE8640 server, supporting four NVIDIA H100 GPUs
• PowerEdge XE9680 server, supporting eight NVIDIA H100 GPUs

In this section, we describe the configuration and connectivity options for NVIDIA GPUs, and how these server-GPU combinations can be applied to various LLM use cases.

NVIDIA GPUs configurations

This design for inferencing supports several options for NVIDIA GPU acceleration components. The following table provides a summary of the GPUs used in this design: 

Table 1. NVIDIA GPUs - Technical specification and use case

GPU connectivity

NVIDIA GPUs support various options to connect two or more GPUs, offering various bandwidths. GPU connectivity is often required for certain multi-GPU applications, especially when higher performance and lower latency are crucial. LLMs often do not fit in the memory of a single GPU and are typically deployed spanning multiple GPUs. Therefore, these GPUs require high-speed connectivity between them.  

NVIDIA NVLink is a high-speed interconnect technology developed by NVIDIA for connecting multiple NVIDIA GPUs to work in parallel. It allows for direct communication between the GPUs with high bandwidth and low latency, enabling them to share data and work collaboratively on compute-intensive tasks.

The following figure illustrates the NVIDIA GPU connectivity options for the PowerEdge servers used in this design:

Figure 2.   NVIDIA GPU connectivity in PowerEdge servers

PowerEdge servers support several different NVLink options:

1. PowerEdge R760xa server with NVIDIA H100 GPUs and NVLink Bridge—NVIDIA NVLink is a high-speed point-to-point (P2P) peer transfer connection. An NVLink bridge is a physical component that facilitates the connection between NVLink-capable GPUs. It acts as an interconnect between the GPUs, allowing them to exchange data at extremely high speeds. 

The PowerEdge R760xa server supports four NVIDIA H100 GPUs; NVLink bridge can connect each pair of GPUs. The NVIDIA H100 GPU supports NVLink bridge connection with a single adjacent NVIDIA H100 GPU. Each of the three attached bridges spans two PCIe slots for a total maximum NVLink Bridge bandwidth of 600 Gbytes per second. 

2. PowerEdge XE8640 server with NVIDIA H100 SXM GPUs and NVLink—The PowerEdge XE8640 server incorporates four H100 GPUs with NVIDIA SXM5 technology. NVIDIA SXM is a high-bandwidth socket solution for connecting NVIDIA Compute Accelerators to a system. The NVIDIA SXM form factor enables multiple GPUs to be tightly interconnected in a server, providing high-bandwidth and low-latency communication between the GPUs. NVIDIA's NVLink technology, which allows for faster data transfers compared to traditional PCIe connections, facilitates this direct GPU-to-GPU communication. The NVLink technology provides a bandwidth of 900 GB/s between any two GPUs.
3. PowerEdge XE9680 server with NVIDIA H100 GPUs and NVSwitch—The PowerEdge XE9680 server incorporates eight NVIDIA H100 GPUs with NVIDIA SXM5 technology. The server includes NVIDIA NVSwitch technology, which is a high-performance, fully connected, and scalable switch technology. It is designed to enable ultrafast communication between multiple NVIDIA GPUs in a single server or across multiple servers in a cluster. NVIDIA NVSwitch facilitates high-bandwidth and low-latency data transfers, making it ideal for large-scale AI and high-performance computing (HPC) applications. The NVSwitch technology provides a bandwidth of 900 GB/s between any two GPUs.

During inference, an AI model's parameters are stored in GPU memory. LLMs might require multiple GPU memory units to accommodate their entire neural network structure. In such cases, it is necessary to interconnect the GPUs using NVLInk technology to effectively support the model's operations and ensure seamless communication between the GPUs. Therefore, the size of the LLM model required by an enterprise dictates which model of PowerEdge server to choose for the inference infrastructure. The following table provides example LLM models that can be deployed in the PowerEdge servers.

Table 2. Example models supported in PowerEdge servers

For more information about generative AI models that were validated as part of this design, see Table 6.

Note: The preceding table does not consider where a model can span multiple nodes (multiple servers) and if the nodes are interconnected with a high-speed network like InfiniBand.