To better understand the sizing recommendations, consider the following key concepts in cnvrg.io:

• Each workspace, such as an interactive Jupyter notebook, runs as a Kubernetes pod that is based on a compute template. 
• A compute template has a predefined set of resources, including CPU, memory, and GPU resources. 
• The pods run on Kubernetes worker nodes in a Kubernetes cluster. 
• The Kubernetes cluster is referred to as a compute resource. 

In this section, we first provide recommendations for creating compute templates that can be used for various AI workloads and use cases. Then, we provide sizing recommendations for different types of worker nodes. Finally, we provide sizing recommendations for the deployment.

The following figure shows the workspaces running on worker nodes using recommended compute templates. In the rest of this section, we describe the sizing of CPU- and GPU-based templates and worker nodes.

Figure 2.    Worker nodes, compute templates, and compute resource in cnvrg.io

Recommended compute templates

In a compute template, the size of the NVIDIA MIG partitions is abstracted, therefore, a template can only specify if it uses a GPU or not. The type of GPU partition that is assigned to a template is based on the worker node (or node pool) associated with the template. These associations are specified using node labels. The following table shows the recommended compute templates that can be created in cnvrg.io:

Table 2.      Recommended compute templates

Note: cnvrg.io deployments come with a default CPU template which has one CPU core and 1 GB memory.

Each compute template can be used for a specific set of workloads or use cases. Our recommended compute templates are categorized into three workload categories:

• CPU Workload templates (including CPU-small, CPU-medium, and CPU-large) can be used for exploratory data analysis and classical machine algorithms that do not require GPU acceleration. For example, these templates are used by data scientists working on problems related to Classification (Logistic Regression, SVM, Decision Trees, XGBoost Classifier, and so on) Regression (Linear Regression, XGBoost Regressor, SGD Regressor) or Clustering (K-Means, DBSCAN, and so on) using the scikit-learn library. The number of CPUs and memory differentiate each template in this workload category. The templates with more CPUs provide the ability to parallelize expensive operations using multiple CPUs, achieving a high degree of parallelization.
• GPU Train Workload templates (including GPU-train-medium and GPU-train-large) can be useful for building and training deep learning (DL) models. Some of the DL problem types include Image Classification, Image Segmentation, Object Detection, Speech Recognition, NLP, and Recommendation. Choosing between a smaller GPU profile and a large GPU profile depends on the model size/parameter count, batch size, and allowable training time. In general, larger GPU profiles provide higher throughput and can support large models such as BERT (NLP), Mask R-CNN (Object Detection), DLRM (Recommendation), and so on. If the model is small, the full computing capacity of the A100 GPUs cannot be used resulting in GPU resource waste. Using the medium template in this case provides a way to maximize the benefits of the GPU such as training a ResNet50 model (Image Classification), SSD (Object Detection), and so on. 
• GPU Inference Workload templates (including GPU-infer-small and GPU-infer-medium) can be useful for inference tasks. Inference tasks can be categorized based on the latency requirements. The smaller template (GPU-infer-small) can be used for online inference in which predictions are generated in real time. The medium template (GPU-infer-medium) can be used for batch inference to generate predictions on a large batch of data in which the latency can be high.

Recommended compute resource 

A compute resource, which is a Kubernetes cluster consisting of Kubernetes worker nodes, runs the cnvrg.io workspaces as pods. We recommend creating different types of worker nodes to support the templates described in Table 2. Worker nodes that have the same configuration are grouped as node pools and are assigned a particular node label.

The following table provides the recommended sizing for different worker node configurations in a Kubernetes cluster:

Table 3.      Recommended sizing for node configurations 

Note that a CPU-only worker node can host multiple CPU-only compute templates. However, each of the GPU worker nodes described in the preceding table can host only one GPU-based compute template. The number of CPU cores and memory allocated to a worker node is more than what is available to the compute template to ensure that enough compute resources are available for system processes such as GPU operators, Kubernetes management pods, and cnvrg.io management pods. 

You must consider several factors for sizing a cnvrg.io deployment, such as the number of projects and workspaces, the type of workspaces, the number of concurrent users, and the number of active tasks. With its modular and microservices-based design, a cnvrg.io deployment can easily be scaled depending on resource use. NVIDIA MIG capability also enables effective partitioning of GPUs for various use cases and increases GPU utilization.

We recommend three sizing deployments for our validated design. The total CPU, memory, and GPU resources for each deployment can be calculated from the number and configuration of worker nodes. The storage recommendation includes storage required for both cnvrg.io control plane and user data. It is assumed that this storage is shared across all the worker nodes. 

The recommended deployments include:

• Minimum production deployment—This deployment is the minimum deployment for a production-ready deployment and is recommended for organizations that are starting their AI journey. It can support up to ten projects and workspaces that perform classic ML or statistical modeling. This deployment assumes that these use cases do not require GPU acceleration and therefore, do not contain GPU worker nodes described in Table 3. You do not need to create GPU-based compute templates that are listed in Table 2 in this deployment. 
• Mainstream deployment—This deployment is recommended for organizations that want to kickstart AI projects that might require GPU acceleration. It can support four projects that require GPU acceleration for training and can host four models for inference. It can also support a few projects and workspaces that use CPU only.
• High-performance deployment—This deployment is recommended for organizations that want to train and deploy AI models at scale. It can support up to 20 active projects that require GPU acceleration for training and can similarly number of models for inference. It can also support several projects and workspaces that use CPU only.

Table 4.      Recommended sizing for the validated design