We conducted performance characterization of the following NeMo models: 345M, 1.3B, 5B, and 20B on a PowerEdge R760xa server with four H100 GPUs connected with NVLink Bridge. We used the Model Analyzer tool, which allowed us to run model sweeps using synthetic datasets. During these sweeps, Model Analyzer performed inference against each NeMo model while varying the number of concurrent requests, enabling us to measure response latencies and gather essential GPU metrics. All NeMo models were run on a single GPU, and Model Analyzer generated comprehensive reports for each of these sweeps.

Model GPU memory consumption

Because GPU memory is a primary bottleneck in the efficient inference of LLM models, it was crucial that we analyzed the model memory consumption. Understanding the GPU memory consumed by various LLM models is pivotal for determining the appropriate resource allocation in production environments. By accurately assessing GPU memory use, we can ensure that the infrastructure is properly sized, avoiding out-of-memory errors and optimizing the overall performance of LLM inference.

Table 9. Model parameter and GPU memory consumption

As a rule, we can estimate that an LLM model consumes approximately 2.1GB of memory for every 1B parameter. However, there are exceptions to this rule, such as smaller models like NeMo 345M. Additionally, non-LLM models like Stable Diffusion have their own memory requirements and might not follow the same pattern. Note that memory consumed by a model might increase slightly as the number of concurrent requests handled by the model increases.

FP16 (Half Precision) and BF16 (BFloat16) are two different floating-point formats used in deep learning and high-performance computing to represent numerical values with reduced precision. Both formats offer benefits in terms of memory usage, computation speed, and energy efficiency. BF16 is typically preferred in deep learning training and inference on specialized hardware like tensor processing units.

Resource use

Using the Model Analyzer report, we can visualize how the latency of the requests change with the concurrent client requests. The following figure shows a graph produced by Model Analyzer for the 20B model:

Figure 12. Model Analyzer report for 20B model

For every model, we identified the optimal number of concurrent requests that minimally impact latency while effectively using available resources. For example, using the preceding graph, we concluded that a single instance 20B model can support a concurrency of 16. This information allows us to achieve the right balance between computational efficiency and responsiveness for each of the NeMo models. The following table summarizes the resource use, latency, and concurrent request metrics for each model for their optimal number of concurrent requests:

Table 10. Resource use and model metrics for optimal number of concurrent requests