Your Browser is Out of Date

Nytro.ai uses technology that works best in other browsers.
For a full experience use one of the browsers below
Design Guide—Generative AI in the Enterprise – Model Customization
Introduction Customizing Large Language Models Solution Components Solution Architecture Validation Results Performance Characterization
Overview
Performance test results
Sizing and scaling guidelines
Dell Professional Services for Generative AI Conclusion References Sample Scripts

Overview

Overview

  • Sizing the infrastructure for LLM customization is essential due to the computational demands, high memory requirements, and unique characteristics of these models and tasks. It is crucial to achieve optimal performance, efficient resource utilization, and reduced downtime. It ensures that the infrastructure can handle the computational and memory requirements of model customization tasks, resulting in faster convergence and reduced operational costs. Properly sized infrastructure also supports scalability for future growth in model size and dataset volume, enhances training time, and maintains consistent quality in fine-tuned models. By preventing underutilization or overutilization of resources, appropriate sizing simplifies management and aligns infrastructure capacity with the demands of model customization. Overall, proper infrastructure sizing for LLM model customization ensures optimal performance, scalability, and user experience while managing operational costs effectively and removing any resource bottlenecks.

    When sizing the infrastructure for model customization, consider several factors to ensure optimal performance and efficiency:

    • Data size and complexity─The size and complexity of the dataset used for model customization can significantly impact infrastructure requirements. Large and complex datasets might demand more computational power and memory. The size of the data is typically measured in tokens or 
    • Model size─The size of the model being fine-tuned or customized is a critical factor. Larger models require more memory and computational resources. Consider the trade-off between model size and performance.
    • Training method─The choice of training method, whether SFT, p-tuning, or LoRA, can affect the infrastructure requirements. SFT is more computationally intensive than PEFT.
    • Batch size─The batch size used during training influences GPU memory requirements. Larger batch sizes typically require more memory but can lead to faster training convergence.
    • GPUs and GPU connectivity─The type of GPU and connectivity between GPUs also dictate the time to train a model. Also, the number of GPUs allocated to the training job determine the time for convergence. In this design, we use a PowerEdge XE9680 server equipped with NVIDIA H100 GPUs and NVSwitch.
    • Parallelization─Implementing data parallelism or model parallelism (tensor parallelism or pipeline parallelism) can distribute the workload across multiple GPUs or nodes. Parallelization can reduce training time but requires infrastructure capable of supporting it. For example, for multinode training, recommend the InfiniBand interconnect between the servers.
    • Convergence time─Consider the acceptable training time and the trade-off between quicker convergence and resource utilization. Shorter training times can be more efficient but might require more powerful hardware.