Maximizing Llama Open Source Model Inference Performance with Tensor Parallelism on a Dell XE9680 with H100s

• 1

  Overview
• 2

  Revisions
• 3

  We value your feedback

• 1

  Scenarios

• 1

  Total Throughput analysis with 2 second TTFT constraint
• 2

  Throughput efficiency analysis with 2 second TTFT constraint
• 3

  Llama-3 70B Throughput analysis without TTFT constraint
• 4

  Llama-3 70B Tokens per second per GPU without any TTFT constraint

• 1

  Total throughput analysis with 2 second TTFT constraint
• 2

  Llama-2 13B TP efficiency analysis with 2 second TTFT constraint
• 3

  Llama-2 13B Throughput analysis without TTFT constraint
• 4

  Llama-2 13B Tokens per second per GPU without any TTFT constraint

• 1

  Conclusion

• 1

  Appendix

• 1

  Dell Technologies documentation
• 2

  Hugging Face and NVIDIA documentation
• 3

  Other documentation