Data exploration

Jupyter Notebook is a platform commonly used by data scientists for exploring data and building a quick proof of concept. In cnvrg.io, we can create a Jupyter Lab workspace type, which allows the user to quickly bring up a Jupyter Notebook workspace. We use the “CPU-large” resource template.

The dataset contains 112,120 frontal-view X-ray images with each image containing one or more of the 14 different thoracic pathology labels. These images are made available as an NFS-cached cnvrg.io dataset. 

Build and train a deep neural network model

Creating a workspace begins by selecting the required compute template, datasets (also data caching to quickly load the data in the notebook), optimized container image, and so on. We use the “GPU-train-medium” resource template, which allocates the grid_a100d-3-40c MIG profile from the A100 80 GB GPU to the workspace. The following figure shows the workspace that we created in cnvrg.io:

Figure 7.   Workspace for AI radiologist

When data exploration is completed and the baseline model is created, the next step consists of performing hyperparameter tuning to optimize the model.

Hyperparameter tuning

Hyperparameter tuning is a critical step in the workflow. There are several ways to perform hyperparameter tuning. It can be as simple as running a grid search algorithm to tune the hyperparameters (more resources might be required) or using complex optimization algorithms such as Bayesian Optimization. Effective hyperparameter optimization also depends on the data scientist’s expertise. 

In cnvrg.io, we can use the Experiments feature that allows data scientists to create parallel jobs to analyze the model metrics and choose the optimized hyperparameters. We used the optimizer as the parameter for turning. Other parameters such as learning rate, batch size, and number of epochs can also be used. The following figure shows the performance of two architectures, ResNet and DenseNet, when turning the optimizer.

Figure 8.   Comparing model performance when tuning hyperparameters

Selecting the best model using flows

We used cnvrg.io flows to evaluate the trained new models to determine the best model for production. The following figure shows the "Eval and Deploy Flow,"  which consists of nine tasks, that we created. The model copy tasks (model_copy_resnet, model_copy_densenet, and model_copy_original) create a copy of the required models from their respective commit to make it available for the flow. This task is performed after analyzing the metrics from the experiments training step. The data tasks (chestxray-pkl-files and chestxray-tfrecords) contain the datasets stored in cnvrg.io and are used by the evaluation tasks. The evaluation tasks (resnet_eval, densenet_eval, and densenet_orig_eval) run an evaluation on validation data and produce the output metric such as the AUC ROC score. The deploy tasks get the artifacts and metrics from the previous tasks, by using the metrics for which we set a condition to select the model based on the maximum average AUC ROC score. The best model is deployed to an endpoint; we can either create a new endpoint or use an existing one. We can send requests with the chest x-ray image file path payload to the serving endpoint and receive the class label predictions as response from the model.

Figure 9.   Flows to evaluate AI models

Deploying the best model

The cnvrg.io Serving feature allows the user to publish the trained model and deploy an endpoint for the prediction. Users can use this endpoint to send requests with the newer unseen chest X-ray images and get the predictions from the model. We use the “GPU-infer-medium” resource template, which allocates the A100-3c-10g MIG profile from the A100 GPU to the workspace.

Figure 10.    Serving models for AI radiologist in cnvrg.io