The following figure shows the architecture for the new generation of the Dell Validated Design for HPC NG-Stor Storage, which uses PowerEdge R660, R760, and R7625 servers and PowerVault ME5084 storage arrays, with the new NG-Stor 6.0 software from our partner company Kalray.
Figure 1. Solution complete architecture
For simplicity and to focus on high-speed links, Figure 1 does not show any management connections (1 GbE) or the external GbE link from the management servers to management customer networks.
Optional PowerVault ME484 EBOD arrays can increase the capacity of the solution. Figure 1 shows such capacity expansion as SAS additions to the existing PowerVault ME5084 storage arrays. PowerEdge R7625 servers, not shown in the figure, are supported but were not evaluated for performance in this release.
The NG-Stor software includes the GPFS as the PFS component that is considered software-defined storage due to its flexibility and scalability. In addition, the NG-Stor software includes many other Kalray software components such as advanced analytics, simplified administration and monitoring, efficient file search, advanced gateway capabilities, and many others.
The main components of the NG-Stor solution include:
- Management servers—PowerEdge R660 servers provide UI, API, and CLI access for configuration, management, analytics, and monitoring of the NG-Stor solution. The servers also perform advanced search capabilities, compiling metadata information in a database to speed up searches and prevent the search from loading metadata Network Shared Disks (NSDs).
- Network Shared Disks (NSDs)—NSDs are backend block devices (that is, RAID 6 LUNs from ME5 arrays or replicated NVMe devices) that store data, metadata, or both. In the NG-Stor solution, file system data and metadata are stored in different NSDs. Data NSDs use spinning media (NLS SAS3 HDDs) or NVMe-replicated pairs. Metadata NSDs use ME5 SSDs in the standard configuration or NVMe devices for high metadata demands (metadata include directories, filenames, permissions, timestamps, and the location of data in other NSDs).
- Storage module—The storage module is the main building block for the NG-Stor storage solution. Each module includes:
- One pair of storage servers
- One, two, or four backend ME5084 storage arrays with optional ME484 capacity expansions
- NSDs that are contained in the backend storage arrays
- Storage server—The storage server is an essential storage module component. HA pairs of PowerEdge R760 servers (failover domains) are connected to ME5084 arrays using SAS 12 Gbps cables to manage data NSDs and provide access to NSDs using redundant high-speed network interfaces. For the standard NG-Stor configuration, these servers are also metadata servers, managing metadata NSDs (using SSDs that replace all spare HDDs in the ME5 arrays).
- Backend storage—Backend storage stores the file system data. The solution uses high-density 5U PowerVault ME5084 (ME5) disk arrays shown in Figure 1. Each ME5 array has two SAS controllers. On the ME5 arrays, two SAS ports from each controller are connected to different HBAs on each of the storage nodes (four SAS cables per server to each ME5084 array). The ME5084 array requires twice the number of SAS cables from each server as previously used by ME4 arrays to match the new ME5 performance. Three cables are required instead of two, however, four cables are used for a better-balanced configuration if there is a failure.
- Capacity expansion storage—Optional PowerVault ME484 capacity expansions (in the lower dotted orange square in Figure 1) are connected behind ME5084 arrays using SAS 12 Gbps cables to expand the capacity of a storage module. For NG-Stor solutions, each ME5084 array is restricted to using only one ME484 expansion for performance and reliability (despite ME5084 arrays officially supporting up to three ME484 expansions).
- High Demand Metadata (HDMD) Server—An HDMD server is a component of the HDMD (in dotted yellow square in Figure 1). Pairs of PowerEdge R660 NVMe servers with up to 14 NVMe devices each in HA (failover domains) provide the metadata NSDs in replicated pairs and supply access to clients using redundant high-speed network interfaces. Other supported servers (PowerEdge R760 and PowerEdge R7625 servers) can be used as NVMe nodes instead of the PowerEdge R660 server.
- NVMe nodes—An NVMe node is the main component of the NVMe Tier Modules (in the dotted red squares in Figure 1). Pairs of the latest PowerEdge servers in HA (failover domains) provide a high-performance flash-based tier for the NG-Stor solution. The NVMe tier consists of three alternatives for PowerEdge servers:
- PowerEdge R660 servers with 14 NVMe direct-attached drives
- PowerEdge R760 servers with 16 NVMe direct-attached devices
- PowerEdge R7625 servers with16 direct-attached drives, all with E3.S PCIe 5 drives
- Native client software—Native client software is installed on the clients to allow access to the file system. The file system must be mounted for access and appears as a single namespace.
- Gateway nodes—The gateway nodes (in the dotted green square in Figure 1) are PowerEdge R760 servers (the same hardware as NG-Hub nodes but using different software) in a Samba Clustered Trivial Data Base (CTDB) cluster providing NFS or SMB access to clients that do not have or cannot install the native client software.
- NG-Hub nodes—The NG-Hub nodes (in the dotted green square in Figure 1) are PowerEdge R760 servers (the same hardware as the gateway nodes but using different software). They offer access to external storage systems (for example, OneFS based storage (PowerScale), object storage (ECS), cloud storage, tape libraries, and so on) allowing them to be used as another tier in the same single namespace using enterprise protocols, including cloud protocols. These nodes can also provide backup, data migration and disaster recovery.
- Management switch—A PowerConnect N2248X-ON Gigabit Ethernet switch connects the different servers and storage arrays. It is used for administration of the solution interconnecting all the components.
- High-speed network switch—Mellanox QM9700 Switches provide high-speed access using InfiniBand (IB) NDR and NDR200. For Ethernet solutions, the Mellanox SN3700 switch is used (SN4700 for 200 GbE).