Our objective is to deploy the Llama 3 70B-Instruct model using the vLLM library on KServe for real-time generative AI inference tasks. The vLLM library provides seamless integration with HuggingFace models and provides an OpenAI-compatible API server. KServe is an inference platform for highly scalable inference. In this validation, we show an example of how to serve the Llama 3 70B-instruct model using vLLM and KServe and send the request to the API endpoint and receive the response generated by the LLM.

Table 8. Software components and versions

For this validation, we installed the vLLM library with ROCm support. The following steps show how to deploy Llama 3 and run inferencing:

1. Create a manifest file for Kind InferenceService, which is required to serve models using KServe:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-70b
spec:
  predictor:
    containers:
    - args:
      - "--port"
      - "8080"
      - "--model"
      - "meta-llama/Meta-Llama-3-70B-Instruct"
      command:
      - "python3:"
      - "-m"
      - "vllm.entrypoints.api_server"
      env:
      - name: HUGGING_FACE_HUB_TOKEN
        value: <UPDATE_HF_TOKEN>
      image: imagehub.ai.lab:5000/corescientificai/private-oci:vllm-0.4.1-rocm6.1-db-v1
      name: vllm-container
      resources:
        limits:
          cpu: "4"
          memory: 600Gi
          amd.com/gpu: "8"
        requests:
          cpu: "1"
          memory: 200Gi
          amd.com/gpu: "8"

2. Create the inference service using the following command:

kubectl create -f kserve-llama3_70B-example.yaml

3. Monitor the Kubernetes resources related to the deployed inference service. Because we do not specify a namespace, create the resources in the default namespace:

kubectl get isvc

kubectl get pods

Wait for the respective isvc resources to become ready; both the containers in the pod must be ready.

4. When the inference service is ready, port forward to test the inference service. Use the name of the pod in the following kubectl port-forward command: 

kubectl port-forward llama3-70b-predictor-00001-deployment-6cf668ccc4-zz7cc 8081:8080

5. From another terminal session, send a request to the model served with KServe to receive the results.

# curl http://0.0.0.0:8081/generate -d '{
"prompt": "The capital of Texas is",
"use_beam_search": true,
"n": 2,
"temperature": 0
}'
{"text":["The capital of Texas is Austin, which is located in the south-central part of the state. Austin is","The capital of Texas is Aus
tin. The city is located in the central part of the state, along the"]}root@omni1:~#

For more information, see the vLLM Meets Kubernetes-Deploying Llama-3 Models using KServe on Dell PowerEdge XE9680 with AMD MI300X blog.