Design Considerations

Physical servers

The database servers tested in this Ready Solution for Oracle—PowerEdge R940-based physical production database servers, a PowerEdge  R740-based physical XVC databases server, and a PowerEdge R940-based virtual databases server—were designed and configured with the following best practices:

• The PCIe network adapters and HBAs that are used for Oracle database public, private interconnect, and SAN traffic were populated based on the recommended PCIe slot priority for optimal power, bandwidth, and thermal performance of the adapters and the system. The populated and the recommended PCIe slots are wired to two separate CPUs in their respective servers, allowing for load balancing of I/Os across the two CPUs.
  • For optimal performance, in the PowerEdge R940-based databases servers, the HBAs were populated in PCIe slots 2 and 5 and the NICs were populated in PCIe slots 3 and 6.
  • For optimal performance, in the PowerEdge R740-based database server, the HBAs were populated in PCIe slots 1 and 7 and the NICs were populated in PCIe slot 3 and the rack Network Daughter Card (rNDC) slot. 
    
    
• The BIOS System Profile was set to Performance.
• Memory DIMMs were populated with at least one DIMM per memory channel across all CPU sockets to maximize the memory throughput with the CPU sockets. The following table shows the capacity and quantity of memory DIMMs populated in each of the database servers that are based on the recommended best practices.

Table 9.           Database servers: Memory DIMMs capacity and quantities

For additional recommended best practices that were implemented on the physical database servers see RHEL 7.4 as bare-metal operating system. 

ESXi host

The virtualized environment in this Ready Solution for Oracle was tested with two virtual databases deployed on a single R940-based VMware ESXi host. 

Figure 6.          Single ESXi host with two VMs running two virtual databases 

As shown in the figure above, the ESXi host contains:

• One virtual machine (VM1) running:
  • RHEL 6.9 as the guest operating system
  • Oracle 11gR2 grid infrastructure software
  • Standalone Oracle Database 11gR2
    
    
• A second virtual machine (VM2) running:
  • RHEL 7.4 as the guest operating system
  • Oracle 12cR2 grid infrastructure software
  • Standalone Oracle Database 12cR2

The ESXi host and the VMs were configured, monitored, and maintained using VMware vSphere Web Client and VMware vCenter Server Appliance (VCSA), which was deployed as a VM on the management server.

In the PowerEdge R940-based virtual databases server, we deployed ESXi 6.4 U2 as the hypervisor.  We applied best practices in our test environment, as described in the following sections.

Note: We used the XtremIO Host Configuration Guide to apply best practices. See this guide for the complete list of best practices for XtremIO storage in a VMware ESXi host environment. 

HBA queue depth settings

The following table lists the default and the recommended HBA queue depth settings in ESXi 6.5 hosts connecting to XtremIO X2 storage arrays:

Table 10.        HBA queue depth settings in ESXi based hosts

We set the LUN queue depth to the recommended value of 256 for the QLogic HBAs used in our virtualized databases server. This setting ensures that the XtremIO X2 storage arrays handle an optimal number of SCSI commands (including I/O requests).

Note: The QLogic HBA Queue Depth setting is no longer read by vSphere, therefore, it is not relevant when configuring a vSphere host with QLogic HBAs.

Multipath configuration

The virtual databases ESXi host was configured by using vSphere Native Multipathing (NMP). The following parameter values are recommended on the host for optimal performance with XtremIO X2 storage:

• Set the native path selection policy on the XtremIO volumes presented to the ESXi hosts to round-robin. 
• Set the vSphere NMP round-robin path switching frequency to XtremIO volumes from the default value (1000 I/O packets) to 1.

Note: In ESXi 6.5, the default path selection policy is round-robin and the default path switching frequency is 1. Therefore, no change was needed in our virtualized databases server.

Host parameter settings

The following ESXi host parameters were configured:

• Disk.SchedNumReqOutstanding—Determines the maximum number of active storage commands (I/O) allowed at any time at the VMkernel. This parameter value for each XtremIO volume presented to the ESXi host was set to the recommended value of 256 by running the following CLI commands on the ESXi 6.5 host:

To get the list of all XtremIO volumes, type:

$> esxcli storage nmp path list | grep XtremIO -B1 | grep "\ naa" | sort | uniq

To set the value for each volume, type:

$> esxcli storage core device set -d <naa.xxx> -O 256

where <naa.xxx> is the XtremIO volume obtained from the previous command.

• Disk.SchedQuantum—Determines the maximum number of consecutive “sequential” I/Os allowed from one VM before switching to another VM. This value was set from its default value of 8 to the recommended value of 64.

Virtual machines

We used the following design principles and best practices to create the VMs in this Ready Solution for Oracle:

• SCSI controllers—We created multiple SCSI controllers to optimize and balance the I/O for the different database disks, as shown in the following table. 

Table 11.        SCSI controller properties set in VMs

• Hard disk drives—All database-related virtual disks—for example, DATA, REDO, FRA, OCR/VD, and TEMP—were assigned the following properties:
  • Type: ‘Thick provisioned eager zeroed’—Which ensures that the space required for the virtual disks are allocated at creation time and the data on the physical device on the storage is zeroed out
  • Sharing: ‘No sharing’—Selected because the deployed databases are standalone databases that do not require sharing database virtual disks with another VM, unlike RAC VMs that share virtual disks between two or more VMs in the database cluster.
    
    
• VM vCPU and vMem—The following table lists the distribution of virtual CPU (vCPU) and virtual memory (vMem) to the two database VMs.

Table 12.     VM configuration: vCPU and vMem details

• Enable disk UUID—In each of the VM options, we added and set the configuration disk.enableUUID parameter to TRUE. This setting ensures that the VMDK always presents a consistent UUID to the VM.

RHEL operating system for Oracle databases

RHEL 7.4 as bare-metal operating system

In the PowerEdge R940-based production database servers and the PowerEdge R740-based XVC databases server, we deployed RHEL 7.4 as the bare-metal operating system. 

Note: We used the XtremIO Host Configuration Guide to apply best practices. See this guide for the complete list of best practices for XtremIO storage in a Linux environment. 

HBA queue depth settings

The following table lists the default and recommended HBA queue depth settings for a Linux environment.

Table 13.        HBA queue depth settings in Linux-based servers

Note: We kept the default queue depth values in our physical database servers because we used Emulex HBAs.

I/O elevator settings

The recommended I/O elevator setting in the RHEL operating system running in the database servers connecting to XtremIO X2 storage arrays is either deadline or noop. The cfg I/O elevator setting is not recommended. In our physical database servers, we used deadline, which is the default I/O elevator setting in RHEL 7.4.

Multipath configuration

We configured the physical database servers using Linux Native Multipathing available in the RHEL 7.4 operating system. We created the configuration file for the /etc/multipath.conf multipath daemon with the following recommended settings:

devices {

device {

                vendor                  XtremIO

               product                 XtremApp

                path_grouping_policy    multibus

               path_checker            tur

               path_selector           "queue-length 0"

               rr_min_io_rq             1

               user_friendly_names     yes

               fast_io_fail_tmo         15

               failback                immediate

       }

Partition alignment in Linux

We partitioned the database disks or XtremIO volumes presented to the Linux-based physical database servers by using fdisk with the default starting sector value of 2,048. This setting ensures that the starting sector number is a multiple of 16 (16 sectors, at 512 bytes each, is 8 KB). Therefore, each database disk is correctly aligned with the XtremIO storage LUN striping.

RHEL as VM guest operating system

In this Ready Solution for Oracle, we deployed RHEL 6.9 as the guest operating system in VM1 running Oracle 11g R2 database and RHEL 7.4 as the guest operating system in VM2 running Oracle 12c R2 database. 

In each guest operating system , we adhered to the following recommended best practices.

PVSCSI driver

For optimal XtremIO X2 storage performance in a VMware environment, we recommend  PVSCSI controllers and driver in the guest VMs. In this Ready Solution for Oracle, we ensured that the inbox RHEL vmw_pvscsi driver module was loaded and used in the guest operating systems.

Note: The PVSCSI driver is used only when the SCSI controller type is set to VMware Paravirtual in the VM settings. 

PVSCSI LUN Queue Depth and ring-pages settings

The following table shows the default and the recommended vmw_pvscsi parameter settings.

Table 14.        PVSCSI parameter settings in guest operating systems

The parameters and their respective recommended values in this table were appended to the kernel boot arguments:

• In the /boot/grub.conf file for the RHEL 6-based guest operating system
• In the /etc/default/grub file for the RHEL 7-based guest operating system

Other guest operating system configurations

Other guest operating systems were configured as follows:

• Multipath was not configured because it is handled at the ESXi host level.
• In the 12c database VM, all database storage disks were set up by using Oracle ASM Filter Driver (ASMFD).
• In the 11g database VM, all database storage disks were set up by using udev rules.