Summary

This Direct from Development (DfD) tech note describes the new AI capabilities you can expect from the Dell PowerEdge R760 and 5th Gen Intel® Xeon® Scalable Processor 8562Y+. Online model serving is critical to enterprises to meet concurrent user needs at low latency, especially with growing model sizes and capabilities. Deploying the latest trend of Generative AI models in production also requires resource management and complex model deployment pipelines. 

In this document, we cover the deployment of two such large language models (LLM) using TorchServe, an open source tool for serving Pytorch models in production. Combined with using Red Hat OpenShift, the industry's leading hybrid cloud application platform powered by Kubernetes, Red Hat OpenShift enables businesses to scale AI workloads easily based on demand.

Market positioning

Red Hat OpenShift (RHOS) provides a robust platform for Large Language Model (LLM) inference and fine-tuning experiments. Leveraging Kubernetes containerization, Red Hat OpenShift Container Platform (RHOCP) packages LLM models and dependencies for easy deployment and portability. Using TorchServe with RHOS, Pytorch models can be deployed within OpenShift clusters to expose model endpoints as RESTful APIs. This enables enterprises to deploy and scale the models of their choice to meet their end-customer or internal users needs for responses at acceptable next-token latencies.

Dell Technologies and Intel® conducted LLM AI performance testing using the latest Dell PowerEdge R760 with 5th Gen Intel® Xeon® Scalable processors. This platform supports up to DDR5-5600 MT/s, thus providing higher memory bandwidth which is critical for LLMs. Support for eight hot-plug 2.5”/3.5” HDD/SSD drives are offered in this platform. Value SAS (vSAS) SSD support has also been expanded to provide more options for an affordable performance SSD tier. These drives can be configured with a Dell rack solution for small businesses that requires a scalable server optimized for enterprise-class workloads.

Dell PowerEdge R760 with 5th Gen Intel® Xeon® Scalable processors also helps "Accelerate Transformation Anywhere" with technology and solutions for AI-driven innovations and automation. 5th Gen Intel® Xeon® Scalable Processors offer advanced CPU capabilities for AI :

• Built in AI accelerator with Intel® Advanced Matrix Extensions (Intel® AMX) built into each core. This accelerator supports BF16 for inferencing and fine-tuning. It also supports quantization with support for INT8 for inferencing, which leads to a significant reduction in memory requirements.
• Intel® Advanced Vector Extensions 512 (Intel® AVX-512) for help with non‒deep learning vector computations.
• Software optimizations for Intel® Xeon® processors are available as part of the latest releases of Intel® Extension for Pytorch GitHub repositories.

System configuration tested

Tables 1, 2, and 3 list details about the hardware platform, software configuration, and workload configuration.

Table 1. Hardware configuration

 Table 2. Software configuration

 Table 3. Workload configuration

Performance results

The test configuration benchmark request spawns a subprocess for each user. Within the subprocess, an independent request is generated and sent to the service endpoint. This endpoint serves as the load balancer and distributes all incoming user requests to existing instances of TorchServe (two instances running per node) using Round Robin logic. The load balancer logic implemented is true round robin using Route exposing. Thus, for every concurrent user test scenario, each instance receives exactly half of the requests. This makes the 2nd token latency the same on both instances (~+/-1ms). Also, the Topology and CPU manager features of OpenShift are enabled to ensure that each instance of TorchServe is affinitized to a single NUMA node. We further leave four CPUs of the total CPUs (128) in the system for CoreOS/other services on the node. With the 124CPUs, we deploy two instances of TorchServe, each with 62 CPUs.

The following figure shows the single node online inference performance for both INT8 and BFloat16 datatypes accelerated using 5th Gen Xeon® built-in AI Acceleration with AMX. With the models in the X-axis and the number of concurrent users supported on the Y-axis, we scale the input token size from 256, 1024. Some of the key takeaways about this graph are:

• LLAMA2-7B and LLAMA2-13B (BF16 and INT8 quantized) models can support up to 64 and 32 concurrent users @ next token latency requirements (~<100ms) for input token size 256, output tokens: 256, Greedy Search.
• LLAMA2-7B and LLAMA2-13B (BF16 and INT8 quantized) models can support up to 32 and 16 concurrent users @ next token latency requirements (~<100ms) for input token size 1024, output tokens: 256, Greedy Search.

From a real-world use case perspective, a two-socket Dell PowerEdge R760, equipped with two Intel® Xeon® Platinum 8562Y+ processors, can handle up to 32 customer chatbot queries, each approximately two pages long. Additionally, it can support up to 64 customer chatbot queries, each around half a page long.

Figure 1. LLM Model Serving using TorchServe on a Dell PowerEdge R760 node with 2x 5th Gen Intel® Xeon® Scalable Processors

Using this methodology, enterprises can meet and support additional concurrent users by scaling out beyond a single node.

Real-world application

One potential way of translating this concurrent user data into a realistic use case is the number of Daily Active Users (DAUs) for a chatbot. Let’s say you support a big company of around 5,000 worldwide workers and are planning to stand up a global chatbot, and you are trying to determine how many Xeon® servers you would need to support it. You start by making some assumptions: that you estimate there will be around 1,500 (DAUs) who will use the tool. You could do the simple math and divide 1,500 DAUs by the number of concurrent users in Figure 1, but you would probably be overprovisioning your Xeon® servers because each of those DAUs would not be hitting the chatbot simultaneously and constantly over a 24-hour period. However, if you assume each DAU engages in five sessions per day with four prompts per session, you should support this chatbot on a single Dell PowerEdge R760 equipped with two Intel® Xeon® Platinum 8562Y+ processors!

How many 5th Gen Intel® Xeon®-based servers does it take to support a chatbot?[1]

Figure 2. Hypothetical example of the number of Daily Active Users for a chatbot that can be supported using a single Dell PowerEdge R760 node with 2x 5th Gen Intel® Xeon® Scalable Processors, based on the concurrency data in Figure 1

Terminology

Table 4. Terminology

Conclusion

This study showcased the performance effectiveness of 5th Gen Xeon® processors on Dell PowerEdge servers for AI Large Language Model (LLM) tasks within the Red Hat OpenShift (RHOS) environment. Specifically, it focused on Meta LLAMA 2 LLM models served using TorchServe supporting concurrent users. Furthermore, the research emphasized the importance of selecting the right combination of server, processor, and software products to achieve scalability and improved performance.

For more details about offline inferencing performance with Intel 5th Gen Xeon® Processors on Dell PowerEdge R760, see this white paper: Driving GenAI Advancements: Dell PowerEdge R760 with the Latest 5th Gen Intel® Xeon® Scalable Processors.

References

• https://pytorch.org/serve/use_cases.html
• https://pytorch.org/serve/server.html
• https://pytorch.org/serve/large_model_inference.html
• https://docs.databricks.com/en/machine-learning/foundation-models/prov-throughput-run-benchmark.html
• https://github.com/intel/intel-extension-for-pytorch/releases/tag/v2.3.0%2Bcpu