VM design and configuration

We used the following design principles and best practices to create the database VMs for the Oracle databases.

SCSI controllers and virtual hard disks

We recommend multiple SCSI controllers of type VMware Paravirtual to optimize and balance the I/O for the different Oracle database hard disks, as described in this section.

The following table shows the recommended SCSI controller design for the OLTP and snapshot database VMs. The DATA and REDO disks are distributed across separate dedicated SCSI controllers because in an Oracle OLTP workload both these types of disks generate a high level of I/O to the storage. In contrast, the OCR, FRA, and TEMP disks generate relatively little I/O and, hence, can exist together on a separate dedicated SCSI controller.

Table 34.        SCSI controller properties in the OLTP and snapshot database VMs

The following table shows the recommended SCSI controller design for the DSS database VM. DSS workloads generate mostly read I/O to the DATA disks with little I/O to the REDO disks. Therefore, as shown in the following table, the 10 DATA disks are distributed across the three dedicated SCSI controllers for load balance, while the rest of the light I/O type disks (the guest operating system, OCR, REDO, FRA, and TEMP disks) exist together on the first SCSI controller.

Table 35.        SCSI controller properties in the DSS database VM

The virtual hard disks for the VMs’ guest operating system are created as a VMFS6 datastore. All the volumes or virtual hard disks for the Oracle databases (DATA, REDO, FRA, TEMP, and OCR disks) are directly added to their respective VMs as raw devices or through raw device mapping (RDM). For these raw devices, although ESXi creates a mapping file with the .vmdk extension and saves it on a VMFS datastore, the mapping file contains only mapping information, while the data itself is stored directly on the storage LUN.

vCPU, vMem, and vNIC configuration

We left all the vCPU and vMem properties in all the Oracle database VMs at their default values except for Memory Reservation. We allocated different quantities of vCPUs, vMems, and memory reservations to the different types of database VMs. For details about the allocated values during the testing of each of the test cases, see Table 2 and Table 7.

We added two virtual network adapters to each of the database VMs: one for in-band VM management and one for Oracle public traffic. We configured the two adapters with the recommended type setting of VMXNet 3. For details about the configuration of virtual switches and physical adapters, see Compute and network design in Appendix B.

Enable disk UUID

For each VM, under VM Options advanced settings, we added the disk.enableUUID configuration parameter and set its value to TRUE. This setting ensures that the VMDK always presents a consistent disk UUID to the VM.

Guest operating system and Oracle ASM configuration

In this reference architecture, we used the following best practices to deploy and configure Red Hat Enterprise Linux 7.4 as the guest operating system in the VMs that were running the standalone Oracle databases:

• Installed and configured the operating system, network, storage disks, Oracle 18c (18.3.0) Grid, and standalone Oracle Database 18c (18.3.0) within the VM, as instructed in the following Dell EMC Knowledge Base article: How to deploy Oracle 18c Grid and Standalone Database on RHEL 7.x.
• Set up the Oracle Grid and database software prerequisites (required operating system RPMs, users, groups, kernel parameters, and so on) by using the information and deployment package in the following Dell EMC Knowledge Base article: Dell EMC Oracle Deployment RPMs for Oracle 18c on RHEL7.x.

We also followed these important best practices:

• Within the guest operating system, for each Oracle virtual disk, we created a single partition that spanned the entire disk and had a starting offset of 2,048 sectors.
• We used UDEV rules to establish ownerships and permissions on the Oracle disks within the VM. The following example shows a UDEV rule set for one of the Oracle disks (REDO disk) within the custom /etc/udev/rules/60-oracle-asmdevices.rules UDEV rules file:

KERNEL=="sd[a-z*[1-9", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="3600601600f004300accaed5bd9741db5", SYMLINK+="oracleasm/disks/ora-redo1", OWNER="grid", GROUP="asmadmin", MODE="0660"

As described in VM design and configuration, we mapped all Oracle database related LUNs that were presented to the ESXi host from the PowerMax storage array directly as raw devices to their respective database VMs using raw device mapping (RDM). In compliance with the Oracle database requirements, we assigned the ownership of the raw devices to the grid user that owns the Oracle GI and Oracle Automatic Storage Management (ASM). The device link for these Oracle related raw devices is /dev/oracles’/disks/oral-XXX. For example, /dev/oracle’s/disks/ora-redo1 is the device link for REDO1 LUN/raw device.

The following table shows the Oracle disk groups that we created for the OLTP database using the raw devices or the virtual disks presented to the VM from the storage array. Except for the OCR disk group that uses the normal redundancy (with triple mirroring), all other disk groups used the external redundancy setting. The coarse striping setting is recommended for DATA, FRA, and OCR disk groups, and the fine-grain striping setting is recommended for REDO1, REDO2, and TEMP disk groups.

Table 36.        ASM disk group design for the OLTP database

Note: The ASM disk group design for the DSS database and the snapshot database is identical to the OLTP database ASM disk group design that is shown in the table with the following exception: In the DSS disk group design, the DATA disk group has eight 500 GB disks for a total disk group size of 4 TB and the TEMP disk group has one 2 TB disk.

Oracle ASM includes a feature through which you can move the data to higher performance tracks of the spinning disks in the compact phase at the end of ASM disk rebalancing. This feature has no benefit for Dell EMC PowerMax storage when physical storage is being virtualized and flash devices are being used. You can disable the rebalancing feature by running the alter diskgroup command for all the disk groups. The following example shows the command for the DATA disk group:

SQL> alter diskgroup DATA set attribute '_rebalance_compact' = 'FALSE';

For more information about ASM compact rebalancing, see Oracle Support note 1902001.1.