NVIDIA AI Enterprise provides enterprise support for various software frameworks, toolkits, and workflows. See the NVIDIA AI Enterprise documentation for more information about all components available with NVIDIA AI Enterprise. The following components incorporated in this validated design are available as part of NVIDIA AI Enterprise:

• NVIDIA NeMo Framework
• Tensor RT-LLM 
• Triton Inference Server
• Base Command Manager Essentials

The following figure shows a block diagram of the NVIDIA Ai Enterprise end to end software suite:

Figure 1.   NVIDIA AI Enterprise

The following sections provide descriptions of each component and how they are used for model customization.

NVIDIA NeMo framework

NVIDIA NeMo is a framework to build, customize, and deploy generative AI models with billions of parameters. The NeMo framework provides an accelerated workflow for training with 3D parallelism techniques. It offers a choice of several customization techniques and is optimized for at-scale inference of large-scale models for language and image applications, with multi-GPU and multinode configurations. The NeMo framework makes generative AI model development easy, cost-effective, and fast for enterprises.

Figure 2. NVIDIA NeMo Framework

Key components of the NeMo Framework include:

• Data curation─NeMo Data Curator is designed to handle large multilingual datasets required for LLMs. It makes tasks like data download, text cleaning, and deduplication easier. This tool uses advanced technologies like MPI, Dask, and Redis Cluster to scale data curation, saving time and effort. Notably, its deduplication feature ensures that language models are trained on unique data, preventing redundancy and potentially reducing costs in the model development phase, making AI more accessible and cost-effective for organizations.
• Distributed training at scale─Training billion-parameter LLM models from the ground up is a complex task that requires powerful distributed computing, advanced parallelism, and various precision techniques. The NeMo framework simplifies this process by efficiently using GPU resources and memory across multiple nodes, significantly reducing training time. It employs parallelism techniques like data, tensor, pipeline, sequence, and sparse attention reduction to optimize training, and it offers precision options such as FP32/TF32, BF16, and FP8. The NeMo framework also includes innovative features like FlashAttention and Rotary Positional Embedding for long sequences, as well as attention with Linear Biases (ALiBi), gradient and partial checkpointing, and the Distributed Adam Optimizer to enhance model performance and speed.
• Support for community-built LLMs─In addition to their own pre-trained models, NVIDIA supports many publicly available models including Llama 2 for model customization. These models need to be converted to NeMo format. The conversion scripts are available as part of the NeMo framework.
• Model customization─The NeMo framework offers an array of techniques to refine generic, pretrained LLMs for specialized use cases. Through these diverse customization options, NeMo offers wide-ranging flexibility that is crucial in meeting varying business requirements. As of the publication of this document, SFT, P-Tuning, and LoRA are supported with Llama 2 models.
• Accelerated inference─NeMo seamlessly works with NVIDIA Triton Inference Server, making AI inference faster and highly accurate with low delays and high throughput. It allows for secure and efficient deployments on single GPUs or large multinode GPU systems while maintaining safety and security standards.
• NeMo Guardrails─NeMo Guardrails plays a pivotal role in upholding the accuracy, appropriateness, relevance, and security of intelligent applications driven by LLMs. It is offered as an open-source solution, encompassing all the essential code, illustrations, and documentation that businesses require to enhance the safety of text-generating AI applications. NeMo Guardrails seamlessly integrates with NeMo and is compatible with all LLMs, including OpenAI's ChatGPT.

The NeMo framework is at the core of this validated design. The framework is made available as two containers through NVIDIA’s NGC catalog: one for training and model customization and another for inference. In this validated design, we used NeMo Framework’s Model Customization “recipes” to customize Llama 2 models that are converted to NeMo format using customization scripts available in the framework on multiple nodes. The customized models were deployed to production using NeMo inference container with Triton Inference Server. See the NeMo User Guide for more information.

TensorRT-LLM

NVIDIA TensorRT-LLM is an open-source library that accelerates and optimizes inference performance of the latest LLMs on NVIDIA GPUs. It lets developers experiment with new LLMs, offering speed-of-light performance and quick customization without deep knowledge of C++ or CUDA.

TensorRT-LLM wraps a deep learning compiler—which includes optimized kernels from FasterTransformer, pre- and postprocessing, and multi-GPU and multinode communication—in a simple open-source Python API for defining, optimizing, and running LLMs for inference in production.

TensorRT is integrated with the NeMo Framework that is used in this validated design for deploying customized Llama 2 models.

Triton Inference Server

NVIDIA Triton Inference Server is inference serving software that standardizes AI model deployment and execution and delivers fast and scalable AI in production. Enterprise support for Triton Inference Server is available with NVIDIA AI Enterprise.

Triton Inference Server streamlines and standardizes AI inference of pretrained and customized models by enabling teams to deploy, run, and scale trained machine learning (ML) or deep learning (DL) models from any framework on any GPU- or CPU-based infrastructure. It provides AI researchers and data scientists the freedom to choose the appropriate framework for their projects without impacting production deployment. It also helps developers deliver high-performance inference across cloud, on-premises, edge, and embedded devices.

Triton Inference Server is integrated with the NeMo framework, ideal for deploying generative AI models. In this validated design, we use Triton Inference Server with the NeMo inferencing container to deploy customized Llama 2 models.

NVIDIA Base Command Manager Essentials

NVIDIA Base Command Manager Essentials facilitates seamless operationalization of AI development at scale by providing features like operating system provisioning, firmware upgrades, network and storage configuration, multi-GPU and multinode job scheduling, and system monitoring. It maximizes the use and performance of the underlying hardware architecture. 

In this validated design, we use the NVIDIA Base Command Manager Essentials for:

• Bare metal provisioning, including deploying the operating system and drivers, and configuring local storage in PowerEdge compute nodes.
• Network configuration, including configuring networks for PXE boot, internal node access, POD networking, and storage networking.
• Kubernetes deployment, including configuring control plane node and worker nodes, access control and provision of Kubernetes management toolkits and frameworks like Prometheus, as well as for NVIDIA software deployment, including deploying NVIDIA GPU operator.
• Slurm cluster deployment including deploying and configuring GPU Direct RDMA and Fabric Manager
• Cluster monitoring and management, including health monitoring, fault tolerance, resource utilization monitoring, software and package management, security and access control, and scaling.