![Banner image](https://cdn-prod.scdn6.secure.raxcdn.com/static/media/c2d8c1ac-a841-482a-867f-c15310c0c4e3.jpeg?_cb=1715976338.651183)
Hugging Face Model Deployments Made Easy by Dell Enterprise Hub
Mon, 20 May 2024 14:46:02 -0000
|Read Time: 0 minutes
Overview
The Dell Enterprise Hub (https://dell.huggingface.co/) is a game changer for obtaining and using optimized models that run on some of the latest and greatest Dell hardware. The Dell Enterprise Hub has a curated set of models that have been containerized with all the software dependencies optimized to run and validated on Dell Hardware.
This blog shows how a user can go from the Dell Enterprise Hub portal to a running model in minutes. We will step through the setup from the beginning until one or more containers are running.
Implementation
The Dell Optimized containers are built on top of the TGI framework (https://huggingface.co/docs/text-generation-inference/index). This allows a user to rely on all the existing benefits of TGI yet it is optimized for Dell. In addition, once a Dell container is downloaded it comes preconfigured with all the required model weights so no additional searching is needed to have a running system. This is a trade-off to have larger containers in order to provide simplicity and minimize accidently running incorrect model weights.
In this blog we look at the simpler case of deploying a model for inference. There are also containers that can be used for model training and fine-tuning and these will be covered in a future blog.
Server setup
During our testing we worked on different Dell Servers and GPUs. In this example we will focus on the 760xa servers for inference.
Hardware
CPU | 2 x Intel(R) Xeon(R) Gold 6438M (32 cores each) |
Memory | 512GB (16 x 32GB) |
Storage | 2TB local storage + PowerScale F600 mounted share |
GPU | 4 x NVIDIA L40S |
This server has the capacity to run multiple inference sessions in parallel. It contains the maximum number of GPUs supported by this Dell server. If more GPUs are required for your model, then an XE9680 can be used that hosts up to 8 GPUs.
Software
The software stack along with the versions we used are below:
- Ubuntu 22.04
- Docker 24.0.6
- NVIDIA Container Toolkit 1.14.2
It's likely that other versions also work but this was what was running in our lab.
Top tip: We missed the install of the NVIDIA toolkit on one server so to avoid this you can run a test container to check if it is working by using the command:
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
If the toolkit is missing, follow the instructions at https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html.
Optimized containers from Dell Enterprise Hub
The optimized containers from Dell Enterprise Hub have three basic requirements - Docker, NVIDIA Container Toolkit, and Linux OS on Dell PowerEdge Platforms.
Select a model
The Dell Enterprise Hub contains an expanding set of models that are optimized to run on Dell hardware. To select a model, go to https://dell.huggingface.co/, log in using your Hugging Face username and select your model of choice. Check out the Model Selection Made Easy by Dell Enterprise Hub blog. It is possible to also use your own fine-tuned model but for this test we will use a prebuilt Llama 3 8B model. For more details on how to use the portal see AI Made Easy Unleash the Potential of Dell Enterprise Hub on Hugging Face.
See below for a sample portal screen for deployment of the Llama 3 8B model on a Dell 760xa with L40S GPUs:
Sample portal screen for Llama 3 deployment on a Dell 760xa with L40s GPUs
See the Dell Enterprise Hub Deploy page for Meta Llama 3 8B instructions.
The models on the Dell Enterprise Hub are under three broad categories of Licenses – Apache 3.0, Llama 3, and Gemma. Even though all of these models are permissive for enterprise usage, you will have to accept terms and conditions before accessing the models.
Container deployment
From the portal above the following Docker run command was generated:
docker run \ -it \ --gpus 2 \ --shm-size 1g \ -p 80:80 \ -e NUM_SHARD=2 \ -e MAX_BATCH_PREFILL_TOKENS=16182 \ -e MAX_INPUT_TOKENS=8000 \ -e MAX_TOTAL_TOKENS=8192 \ registry.dell.huggingface.co/enterprise-dell-inference-meta-llama-meta-llama-3-8b-instruct
This command can be performed as-is on your Dell server and the model will be pulled locally and run.
Note: The containers come prebuilt with all the weights so some models can be more than100GB. This Llama3 8B model is ~27GB.
When running a standard Docker command (as above) there is no link between your Hugging Face account and the model running, this allows a secure access to the models.
To solve for this it is possible to specify your hugging face hub token in 2 ways:
- Set your token as an environment variable “HUGGING_FACE_HUB_TOKEN”
- Add it to each Docker container run command “-e HUGGING_FACE_HUB_TOKEN=$token”
It is important to secure your token and not to post it in any public repo. For full details on how to use tokens see How to use HF tokens and for token generation see Token Generation.
Testing the deployment
The TGI containers expose http endpoints that can be used to perform queries in various formats. The full swagger definition of the API is available at https://huggingface.github.io/text-generation-inference/#/.
For a simple test we can use the “generate” endpoint to POST a simple query to the model that we ran in the previous step:
curl 127.0.0.1:80/generate \ -X POST \ -d '{"inputs":"What is Dell Technologies World?", "parameters":{"max_new_tokens":50}}' \ -H 'Content-Type: application/json'
This produces the following output:
{"generated_text":" Dell Technologies World is an annual conference held by Dell Technologies, a multinational technology company. The conference is designed to bring together customers, partners, and industry experts to share knowledge, showcase new products and services, and network with others in the technology industry.\n"}
As can be seen the response is generated and keeps within the limit of max 50 tokens that was specified in the query.
Conclusion
The Dell Enterprise Hub simplifies the deployment and execution of the latest AI models. The prebuilt containers run seamlessly on Dell Hardware.
In this example we showed how quick and easy it is to run the latest Llama 3 model on a 760xa with L40S GPUs. The Dell Enterprise Hub also supports training and fine-tuning models on Dell Hardware.
Eager for more? Check out the other blogs in this series to get inspired and discover what else you can do with the Dell Enterprise Hub and Hugging Face partnership.