Your Browser is Out of Date

Nytro.ai uses technology that works best in other browsers.
For a full experience use one of the browsers below
Design Guide—Generative AI in the Enterprise - Inferencing
Introduction AI Model Inferencing Architecture Concepts and Components Solution Architecture Validation Results Performance Characterization and Sizing Inferencing Operations Dell Professional Services for Generative AI Conclusion
Summary
We value your feedback
References

Summary

Summary

  • The Dell Validated Design for Generative AI Inferencing with NVIDIA has been developed to address the needs of enterprises that need to develop and run custom AI LLMs using domain-specific data that is relevant to their own organization.

    Dell Technologies and NVIDIA have designed a scalable, modular, and high-performance architecture that enables enterprises to more quickly design and deploy an inferencing solution that has been validated and performance-tested to accelerate the time to value and to reduce the risk and uncertainty by using a proven design.

    This guide provides design guidance and a fully validated reference architecture for generative AI inferencing. Topics that were discussed include:

    • The definition of inferencing and how it fits into the AI model development life cycle
    • Use cases for inferencing, including some of the challenges in implementation
    • Explanation of LLM characteristics and examples
    • Details about the Dell PowerEdge servers and NVIDIA GPUs used in the design, including GPU configurations, GPU connectivity, and networking methods.
    • Descriptions of the primary software components used for inferencing, including NVDIA AI Enterprise, Triton Inference Server, the NeMo framework for generative AI models, and the NVIDIA cluster manager
    • A detailed description of the reference architecture for generative AI inferencing, including both the physical hardware and the software architecture
    • A presentation of the validation results, including the numerous models used for validation and the multiple validation scenarios
    • A listing of the performance test results and how they influence the infrastructure sizing recommendations
    • Guidelines for inferencing operations when the solution is deployed and operational
    • Descriptions of the Dell professional consulting services that have been designed specifically for this validated design for generative AI, including advisory, implementation, adoption, and scaling services

    While this design focuses on AI inferencing of pretrained models, it is the first in a series of validated designs for generative AI that focus on all facets of the generative AI life cycle, including inferencing, model customization, and model training. While these designs focus on generative AI use cases, the architecture is more broadly applicable to more general AI use cases as well.

    With this project, Dell Technologies and NVIDIA enable organizations to deliver full-stack generative AI solutions built on the best of Dell infrastructure and software, combined with the latest NVIDIA accelerators, AI software, and AI expertise. This combination of components enables enterprises to use purpose-built generative AI on-premises to solve their business challenges. Together, we are leading the way in driving the next wave of innovation in the enterprise AI landscape.