Figure 6 shows our modeling workflow. To generate the performance models, we begin by collecting the performance samples by instrumenting the application under consideration and benchmarking the application for various performance parameters to collect the performance samples (Step 1 in Figure 6). The performance metric under consideration could be execution time, total number of FLOPs, or in this case, peak memory consumption during execution. Upon collecting the performance samples, the data is classified into training data and test data through sampling (Step 2). The training data is used during model generation process. For the model generation in our study, we use Pandora , a symbolic regression based tool built in Python. The tool leverages several data structures and algorithms from the DEAP evolutionary algorithm library , which were extended with numerous optimizations for symbolic regression. While the original developers developed Pandora for the approximate computing applications , we leveraged the framework for performance and peak memory consumption modeling. In a previous work, we performed extensive evaluation on Pandora for performance modeling by building accurate coarse-grained multi-parameter models for various HPC applications . More details about the Pandora tool are given in Pandora – symbolic regression based modeling tool. When the performance model has been generated, the next step in the workflow is to use the test/validation data to evaluate the accuracy of our model (Step 4). In Model evaluation, we demonstrate the accuracy of our modeling approach by generating a memory consumption model of 3D U-net to predict the peak memory consumption during the training phase.