Your Browser is Out of Date

Nytro.ai uses technology that works best in other browsers.
For a full experience use one of the browsers below
Implementing a Digital Assistant with Red Hat OpenShift AI on Dell APEX Cloud Platform for Red Hat OpenShift
Introduction Business and technical challenges Solution benefits Partner technology overview Solution overview Solution architecture Model serving
Llama 2 deployment
Conclusion References

Llama 2 deployment

Llama 2 deployment

  • Deploying a LLM is a multi-step process which can be challenging. A robust AI/ML platform requires the right combination of hardware and software.

    Dell APEX Cloud Platform for Red Hat OpenShift reduces this complexity by providing an AI/ML solution for data scientists and data engineers to seamlessly deploy a LLM model.

    In this exercise, we deployed a LLM based digital assistant on Dell APEX Cloud Platform for Red Hat OpenShift that can answer user queries related to domain specific documents. As text-based searches are limited in obtaining the right data, a digital assistant can help retrieve more accurate and relevant results using semantic search and natural language processing.

    The following list describes the AI components used in this exercise.

    • Llama 2: A second generation, open-source, pre-trained, and fine-tuned LLM ranging from 7B to 70B parameters. It can be used to build chatbots, language generation, and other AI-powered tools.
    • RAG: An AI framework that combines pre-trained language models with a retrieval mechanism. It acts as a bridge between the language models and a repository with large amounts of data, helping LLMs provide better and more accurate answers.
    • LangChain: A framework for developing applications powered by language models. It is developed to simplify the process of building LLM powered applications by providing an abstracted standard interface that makes it easier to interact with different language models, including Llama 2.
    • Gradio: Gradio is an open-source Python library that enables incredibly fast development and prototyping of ML web applications with user interfaces. It provides a simple and intuitive API which is compatible with all Python programs and libraries. Gradio provides a variety of options to customize various elements of the user interface (UI).
    • Redis: Redis is a popular in-memory data structure store. One of the features of the Redis database is the ability to store embeddings with metadata for LLMs to use. Redis vector database is an excellent choice for applications that need to store and search vector data quickly and efficiently.

    Combining these AI technologies with Dell APEX Cloud Platform for Red Hat OpenShift and Red Hat OpenShift AI offers organizations an efficient way to deploy an integrated platform for AI/ML workloads. This solution mitigates prior complexities and enables data scientists to efficiently develop, deploy, and manage the full life cycle of data science and ML workloads.