Your Browser is Out of Date

Nytro.ai uses technology that works best in other browsers.
For a full experience use one of the browsers below
Design Guide—Generative AI in the Enterprise - Inferencing
Introduction AI Model Inferencing Architecture Concepts and Components
Introduction
Large language models
Dell PowerEdge Servers and NVIDIA GPUs
Inferencing software components
Solution Architecture Validation Results Performance Characterization and Sizing Inferencing Operations Dell Professional Services for Generative AI Conclusion References

Introduction

Introduction

  • In this chapter, we explain some of the key architectural concepts that are part of this design for generative AI inferencing, including LLM characteristics and examples. We list the Dell PowerEdge servers and NVIDIA GPUs used in the design, including GPU configurations and GPU connectivity and networking methods.

    We also describe the primary software components used for inferencing, including NVDIA AI Enterprise, Triton Inference Server, the NeMo framework for generative AI models, and the NVIDIA cluster manager.