The GenAI solution is constructed by integrating essential components, each contributing to its robust foundation:
- Dell APEX File Storage establishes a sturdy data foundation to store the raw, training, testing and inferencing datasets.
- AWS acts as the infrastructure provider, hosting and managing the Databricks clusters and Dell APEX File Storage.
- Databricks operates as the distributed computer cluster and environment, dedicated to tasks such as training, fine-tuning, and inferring LLM models.
- For model resources, both Hugging Face's transformer and MosaicML's open-source Streaming Dataset, along with Composer libraries, offer pretrained LLM models and Synthetic Datasets for fine-tuning. Also, they provide tools to refine raw data sourced from Dell APEX File Storages, enabling efficient model fine-tuning and inferencing.