This section covers the LLM model preparation required before deploying it in the Caikit model serving environment. For the model to be served using Caikit, it must be converted to Caikit format before deployment.

Note: Obtain the required access for Llama 2 from the Meta webpage before proceeding with the following steps.

Convert Llama 2 in Caikit format

The following steps describe the process to convert Llama 2 in Caikit format. For more information, see GitHub.

1.  Download and install git-lfs.

!wget https://github.com/git-lfs/git-lfs/releases/download/v3.4.0/git-lfs-linux-amd64-v3.4.0.tar.gz
!tar -xvzf git-lfs-linux-amd64-v3.4.0.tar.gz
!PREFIX=/opt/app-root/src/.local ./git-lfs-3.4.0/install.sh 

2. Clone Caikit nlp repository.

!git clone https://github.com/caikit/caikit-nlp.git

3. Install caikit-nlp.

!pip install ./caikit-nlp

4. Clone caikit tgis serving repository.

!git clone https://github.com/opendatahub-io/caikit-tgis-serving.git

5. Copy model conversion python script "convert.py" to the current location.

!cp caikit-tgis-serving/utils/convert.py . 

6. Install and upgrade huggingface_hub module and log in to your huggingface account.

Note: Make sure you have your HUGGINGFACE_USER and HUGGINGFACE_TOKEN added as environment variable of this notebook

!pip install --upgrade huggingface_hub
!huggingface-cli login --token $HUGGINGFACE_TOKEN

7. Download Llama-2-7b-hf model from Hugging Face.

!git clone https://${HUGGINGFACE_USER}:${HUGGINGFACE_TOKEN}@huggingface.co/meta-llama/Llama-2-7b-hf

8. Convert Llama 2 model to Caikit format. Your converted model will be saved to Llama-2-7b-hf-caikit directory.

!./convert.py --model-path ./Llama-2-7b-hf/ --model-save-path ./Llama-2-7b-hf-caikit

Upload Model to Object Storage

Use the following steps to upload the model to Object Storage (GitHub).

1. Install boto module and libraries if it does not exist in your current session.

#! pip install --upgrade pip
#! pip install boto3 botocore

2. Create helper functions for easier uploading and listing model. 

Note: Make sure you have the following environment variables set for connection to your object storage.

• AWS_S3_ENDPOINT
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION
• AWS_S3_BUCKET

import os
import boto3
import botocore
 
aws_access_key_id = os.environ.get('AWS_ACCESS_KEY_ID')
aws_secret_access_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
endpoint_url = os.environ.get('AWS_S3_ENDPOINT')
region_name = os.environ.get('AWS_DEFAULT_REGION')
bucket_name = os.environ.get('AWS_S3_BUCKET')
 
session = boto3.session.Session(aws_access_key_id=aws_access_key_id,
                                 aws_secret_access_key=aws_secret_access_key)
 
s3_resource = session.resource(
    's3',
     config=botocore.client.Config(signature_version='s3v4'),
    endpoint_url=endpoint_url,
    region_name=region_name)
 
bucket = s3_resource.Bucket(bucket_name)
 
 
def upload_directory_to_s3(local_directory, s3_prefix):
    for root, dirs, files in os.walk(local_directory):
        for filename in files:
            file_path = os.path.join(root, filename)
            relative_path = os.path.relpath(file_path, local_directory)
            s3_key = os.path.join(s3_prefix, relative_path)
            print(f"{file_path} -> {s3_key}")
            bucket.upload_file(file_path, s3_key)
 
 
def list_objects(prefix):
    filter = bucket.objects.filter(Prefix=prefix)
    for obj in filter.all():
        print(obj.key)
     

3. Upload your model directory to object storage.

 Note: Do not upload your model to root of the bucket

upload_directory_to_s3("Llama-2-7b-chat-caikit", f"example-models/llm/models/Llama-2-7b-chat")

4. List uploaded models.

list_objects(f"example-models/llm/models/Llama-2-7b-chat")