The current intersection of high-performance computing (HPC), artificial intelligence (AI), and data analytics workloads stems from the convergence of workflows that require different techniques to solve complex problems. AI and data analytics are being used to augment traditional HPC workloads. Combining AI and data analytics methods with traditional HPC workflows can speed scientific discovery and innovation processes. Data scientists and researchers are developing new processes for solving problems at massive scale, requiring compute resources such as those of HPC systems. AI and data analytics workloads benefit from an HPC infrastructure that can scale up to improve performance.
This convergence of workloads is causing customers to look for a unified architecture that supports all the workloads. Traditional HPC workloads such as simulation and modeling are compute-intensive and benefit from fast interconnects and high-performing file systems. Traditional HPC workloads are submitted through a batch scheduler, taking hours or days to run. However, AI and data analytics workloads are data-intensive and require tools from data ingest to data science workbenches. AI and data analytics workloads are interactive; data scientists are iterative when building and training models. Typically, AI and data analytics workloads are initiated and terminated repeatedly, depending on the development stage of the data workflow.
The variance in HPC, AI, and data analytics workloads can lead customers to believe that they need three separate environments; however, this belief is not accurate. Customers can design a unified architecture with multipurpose, balanced nodes to support all workloads. Also, with the correct software and tools, customers can satisfy traditional HPC users as well as AI and data analytics users without forcing them to learn new skills and adjust to a new operational model.
Integrating all three workloads on a single architecture presents challenges, however. Customers must consider that:
The Dell EMC HPC Ready Architecture for AI and Data Analytics takes a building-block design approach, giving customers flexibility in architecture choices that best meet their environmental needs. The architecture design encompasses:
The HPC Ready Architecture for AI and Data Analytics provides an enterprise supported version that uses Bright Manager and Bright Cluster Manager for Data Science.
Bright Cluster Manager for Data Science enables you to manage data science clusters as a single entity. You can provision the hardware, operating system, HPC, big data software, and deep learning software from a single interface. For customers who want an open-source option, Dell EMC provides Ansible playbooks that provide a convenient way to install the software for the HPC Ready Architecture for AI and Data Analytics.