Follow the instructions below to setup benchmark environment in VMs:
- Based on Appendix A, use the same VM images for benchmarking.
- Define VMs with the following commands (XML definitions are in Appendix D):
- virsh define tdx.xml
- virsh define no-tdx.xml
- Start VM with virsh start command and login to it over SSH.
- Prepare benchmark environment using point 1, 2b, and 3 from instructions:
https://github.com/intel/intel-extension-for-pytorch/tree/v2.1.100%2Bcpu/examples/cpu/inference/python/llm#environment-setup - Request access to Llama 2 model from: https://huggingface.co/meta-llama/Llama-2-7b-hf
- Generate Hugging Face token from: https://huggingface.co/settings/tokens
- Log into Hugging Face with token in VM:
- Run benchmark with commands and adjust parameters to your needs:
- For int8 precision:
- For bf16 and fp32 precision: