LAN setup

Physical network design and connectivity

The following figure shows the solution’s redundant and highly available physical LAN design and connectivity.

Figure 7.         Physical network design and connectivity

Note: The ports on the switches shown in the preceding figure, to which the database server network ports are connected, are for illustration purposes only. Network administrators can choose any available ports on the switches, as appropriate.

The four database servers tested in this solution were configured with a highly available network:

• In the two physical production RAC database servers represented by R940-Prod-DB-Node1 and R940-Prod-DB-Node2 in Figure 7.
  • Each RAC node has two 10 GbE network ports for Oracle public traffic connected to two separate ToR switches.
  • Each RAC node has two 10 GbE network ports for Oracle private interconnect traffic connected to two separate ToR switches.
  • Each RAC node has two 1 GbE network ports connected to the management switch. One port is for server management from within the operating system and one port is for out-of-band management of the server by using iDRAC.
    
    
• In the single physical XVC database server represented by R740-XVC-DB-Server in Figure 7:
  • Two 10 GbE network ports for Oracle public traffic are configured and connected to two separate ToR switches.
  • Two 1 GbE network ports are configured and connected to the management switch. One port is for in-band server management from within the operating system and one port is for out-of-band management of the server by using iDRAC.
    
    
• In the single virtualized database server represented by R940-Virtual-DB-Server in Figure 7:
  • Two 10 GbE network ports for Oracle public traffic are configured and connected to two separate ToR switches.
  • Two 1 GbE network ports are configured and connected to the management switch.  One port is for in-band server management using the VMware vCenter Server Appliance and one port is for out-of-band management of the server by using iDRAC.

Oracle public networks from all database servers and private interconnect networks from the two Oracle RAC nodes are configured on the same redundant ToR 10 GbE S4048-ON switches. However, the network traffic is segregated by using VLANs as shown in the following table.

Table 15.        Sample VLAN configuration on S4048-ON 10 GbE ToR switches

Note: VLAN IDs used in the table are examples. Network administrators can use VLAN IDs that conform to their network policies and standards only if the Oracle public and private networks are on two separate VLANs.

Virtual network design

The virtual environment in this Ready Solution for Oracle was tested by using a single VMware ESXi host running two  VMs.  One VM for running a standalone Oracle 11gR2 database and the other VM for running a standalone Oracle 12c R2 database. 

Figure 8.         Virtual network design on the VMware ESXi database host

The preceding figure shows the virtual or VM network topology implemented in the virtualized databases ESXi host. It also shows how the virtual layer maps to the physical layer.

As shown in the figure, the VMware-based virtual network design in the Ready Solution for Oracle consists of the following virtual switches and port groups:

• Public distributed switch—This switch is implemented as a distributed virtual switch for Oracle public traffic, which is generated by the two virtual databases running inside two separate VMs. In this distributed switch, we created one distributed port group and two uplink ports:
  • Public distributed port group—This group provides the virtual interfaces for Oracle public traffic for the two Oracle database VMs, which are represented by the VM names xorb-virt-11g and xorb-virt-12c in the figure. This port group is tagged with VLAN ID 16, which is the same as the VLAN ID that is configured on the S4048-ON ToR switch for public traffic. 
  • Two physical uplink ports—Two 10 GbE physical ports from two separate NICs serve as uplink ports to the public distributed switch to provide sufficient bandwidth and redundancy.
    

NOTE: Typically, for single ESXi host implementations, standard switches and port groups are sufficient. Distributed switches and port groups configured in this solution provide easy expansion to virtual Oracle RAC environments by using multiple ESXi hosts, if needed.

• Management standard switch—This switch is implemented as a standard switch for management traffic. The default VMKernel port in the Management Network port group, vmk0, is used to manage the ESXi host from VMware vCenter. The VM Network port group provides the virtual interfaces to access and manage the Guest VMs. Both these management traffic interfaces use one 1 GbE LAN On Motherboard (LOM) port in the ESXi host that is connected externally to the S3048-ON management switch. 

NOTE: Though not shown in Figure 8 or implemented during testing of this Ready Solution for Oracle, we recommend that you configure an additional 1 GbE management network port for redundancy.

SAN setup

We used the following recommended SAN connectivity and zoning best practices to configure all the database servers with the XtremIO X2 storage arrays:

• Use at least two initiators per ESXi host or physical database servers for load balance and bandwidth. For high availability, place the two initiators on separate HBAs.
• Implement redundant FC switches for high availability.
• Ensure that one FC zone set includes one HBA port or initiator and at least one FC target port on each XtremIO storage controller.

The following figure shows the recommended physical SAN connections between the database servers, the FC switches, and the dual X-Brick XtremIO X2 cluster storage arrays in this Ready Solution for Oracle.

Figure 9.         SAN setup: Physical design and connectivity

Note: The ports on the FC switches shown in the figure to which the database server HBA ports and the XtremIO FC front-end ports are connected are for illustration purposes only. SAN administrators can choose any available ports on the switches, as appropriate.

The SAN connectivity and redundancy of the components in the solution ensure that no single point of failure exists and provides the necessary bandwidth. As shown in the figure, the SAN setup in the Ready Solution for Oracle with XtremIO X2 storage consists of:

• HBAs in the database servers:
  • Two dual-port 16 Gb/s HBAs per database server with a total bandwidth of 8 GB/s per server. Two dual-port HBAs per node provide HA and bandwidth if one HBA fails. 
  • Each port on the same HBA adapter is connected to two separate FC switches to provide redundant paths and HA if one FC port or FC switch fails.
    
    
• Redundant FC switches:
  • Two 16 Gb/s FC switches provide redundant paths for bandwidth and connectivity between the database servers and the storage controllers if one FC switch fails. 
    
    
• Front-end FC ports in the XtremIO storage controllers:
  • Two 16 Gb/s FC front-end ports per XtremIO I/O storage controller with a total bandwidth of 8 GB/s per controller.
  • Each FC front-end port per controller is connected to two separate FC switches to provide redundant paths and HA if one or more FC front-end ports, one or more storage controllers, or one FC switch fails.

Zoning overview

The following figure shows the logical view of the Oracle RAC production database servers after the recommended zoning configurations are created on the redundant FC switches. The same design and configuration is used for the XVC database and the virtualized database servers. Zoning is configured so that each host initiator in the database server is zoned to four target front-end ports that are located on four separate XtremIO storage controllers. This configuration provides a total of 16 paths per database server and ensures sufficient bandwidth and availability for the Oracle database servers to reach the storage if one or more ports or HBAs, a switch, or storage FC ports or controllers fail.

Figure 10.     SAN setup: Zoning logical view

DD6300 network design

To test the backup and recovery solution, we connected the DD6300 system as the backup appliance to the Ready Solution for Oracle with XtremIO X2 storage. As shown in the preceding figure, we connected two 10 GbE ports from two separate NICs on the DD6300 system to two separate S4048-ON 10 GbE switches. S4048-ON switches serve as the ToR Ethernet switches for Oracle database public and private interconnect network traffic in this Ready Solution for Oracle.

Figure 11.     Data Domain DD6300 backup and recovery solution: IP network connections

For the databases in this Ready Solution for Oracle to communicate with the DD6300 backup appliance, the two DD6300 network ports on the S4048-ON switches were added as untagged 10 GbE ports to VLAN (configuration) ID 16. This VLAN serves as the public VLAN in this Ready Solution for Oracle. We connected the management port on the DD6300 system to the S3048-ON 1 GbE switch that serves as the management switch in the Ready Solution for Oracle with XtremIO X2 storage.

In the DD6300 system, we configured the two network ports with static IP addresses that belong to the same subnet as the Oracle database public network in this Ready Solution for Oracle. These two network interfaces were added to the default interface group under the DD Boost protocols configuration.