Intel Optane PMem operating modes

In PMem App Direct Mode (PMem-AD), the data stored on PMem is persistent across power cycles like any other persistent storage media. Any DRAM installed in the server is seen as the main memory by the operating system and by any application (like the database) installed in the system. For an operating system to access PMem that is configured in App Direct Mode in the server’s Unified Extensible Firmware Interface (UEFI), the operating system requires that it contain the specific PMem kernel driver library or modules. An application (like a database) installed in the system, in turn, can access the PMem devices in multiple ways—either as byte-addressable persistent memory or as block-addressable persistent storage—depending on how the PMem devices are configured at the operating system level. These two access methods of PMem-AD are described further in the following sections.

Persistent Memory over App Direct (byte-addressable)

As shown in the following figure, to configure and access PMem directly as a byte-addressable device (for example, using it as a direct load/store device like main memory) requires that:

• The operating system contains the PMem driver module and supports a direct-access (DAX) file system (fsdax or ext4), and
• The application code is modified to access PMem directly over a DAX file system.

Figure 2.              PMem – Persistent Memory (byte-addressable) over App Direct Mode

This is the fastest way for an application to access the PMem devices and to get the most out of PMem’s performance capability. This is because it completely by-passes the operating system’s I/O stack (page cache and block layer). This way of directly accessing the data on PMem also eliminates the need for an application to copy the data into buffers in DRAM. For example, for Oracle Databases, the buffer cache component of the SGA would be considered to be the database buffer in DRAM. The ability for the database to directly fetch the data from PMem would eliminate the need to copy the data first into the buffer cache in DRAM.

At the time of this writing, Oracle Database 19c and older versions do not support the direct-access (DAX) feature. Hence, PMem as a persistent memory over App-Direct Mode is not yet a candidate to be used with Oracle Databases for storing datafiles.

Persistent Storage over App Direct (block-addressable)

Those applications that have not yet been modified to directly access the PMem devices as byte-addressable can still access it as a traditional block-addressable persistent storage device. Accessing it as a block storage, however, still requires that the underlying operating system contains the necessary PMem driver module. PMem can be accessed as a persistent block storage by configuring it in either ‘raw’ namespace or in ‘sector’ namespace mode at the operating system level, as shown in the following figure. Sector mode is also called legacy mode and uses the standard file API.

Figure 3.              PMem – Persistent Storage (block-addressable) over App Direct Mode

PMem is fundamentally a byte-addressable device. In ‘raw’ mode, although the application accesses the PMem as a block device, underneath, the base PMem driver still accesses and operates on the data at a byte-level. PMem in ‘raw’ mode can guarantee a power-fail atomicity[1] of only eight bytes. For applications like Oracle Databases that operate in chunks of 8,192 bytes (8 K blocks), the ‘raw’ mode exposes them to the risk of ‘torn writes’ which could result in potential data loss or inconsistency in the event of a power failure.

To mitigate this ‘torn writes’ problem, the ‘sector’ mode uses the Block Translation Table (BTT) driver, which implements the copy-on-write optimization on top of the PMem driver, as shown in the figure above. This helps to provide power-fail atomicity and lowers the risk of data loss or inconsistency. The ‘sector’ mode is also called the legacy mode.

For further details on PMem access modes, atomicity and the ‘torn writes’ problem, see the Oracle blog https://blogs.oracle.com/database/persistent-memory-primer.

Since Oracle Database 19c and older versions do not yet support the direct-access (DAX) feature, we tested the PMem as a persistent block-addressable storage (legacy mode) over App Direct Mode. In legacy mode, PMem as a persistent block storage was used and tested as the In-Memory FastStart (IMFS) disk area. The details of this testing are described in Use Case 2: IMCS load performance with and without IMFS area.

At the time of this writing, storing Oracle Database 19c datafiles on Intel Optane PMem as persistent storage (running in App Direct Mode) is NOT supported, including using it as the IMFS disk area. Therefore, this design guide focuses mainly on how to configure PMem modules as a persistent block-addressable storage over App-Direct mode with Oracle Databases and places less emphasis on its performance. This is described in Use Case 2: IMCS load performance with and without IMFS area section.

For further details on Intel Optane PMem, see the Quick Start Guide: Provision Intel Optane Persistent Memory.