Home > Storage > Unity XT > Virtualization, Cloud & Applications > Dell Unity XT: Oracle Database Best Practices > Shared subnets
If multiple network paths are needed for dNFS/kNFS control traffic and dNFS data traffic, Oracle recommends that each path uses a separate subnet. However, using separate subnets may not always be possible.
This section discusses the use of subnet sharing on network paths for Oracle dNFS.
When configuring dNFS to use a subnet that is shared with another network interface, additional configuration, including IPv4 network routing filter and static routing, is required.
According to Oracle, if any IP address used by dNFS is from a subnet on another NIC interface, the following additional configuration changes must be made:
Note: For more information about dontroute and oranfstab, see Oracle Database—Database Installation Guide.
For more information about shared subnets, see
Linux 8 follows the recommendations of ingress filtering for multihomed networks (http://tools.ietf.org/html/rfc3704). These routing filters must be relaxed for multiple NIC interfaces in the same server to use the same subnet. Before configuring network interfaces to use the same subnet, ensure that you relax routing filters.
Before installing Oracle, if the Oracle 21c preinstall rpm is used to configure the operating system, routing filters will be relaxed appropriately.
With OL 8u8, rp_filter is set to 2 for all network interfaces including the default interface. A value of 2 (loose filtering) is acceptable. Should no filtering (rp_filter = 0) or strict filtering (rp_filter = 1) be needed, see Oracle Grid Infrastructure—Grid Infrastructure Installation and Upgrade Guide, and My Oracle Support Doc ID 1286796.1.
If dNFS multipathing is needed and there is only one subnet, loose filtering (rp_filter = 2) needs to be set.
Setting rp_filter to an incorrect value can cause interconnect packets to be blocked or discarded in a RAC configuration.
rp_filter value | Description |
0 | No filtering |
1 | Strict filtering |
2 | Loose filtering |
To verify that IPv4 routing filters have been relaxed, look at the rp_filter settings in /etc/sysctl.conf. The values should be 2.
If IPv4 reverse path filtering is not set, add or change the IPv4 rp_filters in /etc/sysctl.conf and make them active.
Starting with OL kernel 2.6.31, Oracle RAC systems that use multiple NICs for the private interconnect must have parameter rp_filter set appropriately.
For information about channel bonded interfaces for NFS control traffic, see Oracle MOS notes:
When dNFS traffic must route traffic to an IP address from a subnet already in use on the NFS client, static routing must be configured. A static route must be defined for each of the network addresses from a shared subnet and used by dNFS as a route. This requirement includes defining a static route for dNFS data traffic should it use the same subnet as dNFS control traffic. If static routes are not defined, automatic load balancing and performance tuning of dNFS will not operate as expected per the dNFS path definitions in file oranfstab.
The remainder of this section provides two examples of static routing. The first example considers two interfaces sharing a subnet, and the second example considers four network interfaces sharing a subnet.
The following figure illustrates a scenario where an NFS client has two interfaces. Interface enp132s0f0 is intended to be dedicated to dNFS traffic, and interface eno3 is for general public Ethernet traffic. Both interfaces use a network IP address from the same subnet, and default routing is defined. Figure 38 shows dNFS traffic flowing through interface eno3 rather than the intended interface enp132s0f0 when default routing is used.
The following two figures show the corresponding routing table for Figure 38.
If default routing and IP addresses from the same subnet are used, the operating system searches the routing table for the route that best matches the destination address and mask. The operating system will use the best-matched route found. Because interfaces eno3 and enp132s0f0 share the same subnet (nnn.nnn.nnn.nnn) and CIDR (cc), the operating system considers both entries as best-matched. When multiple best-matched entries exist, the operating system selects the top-most best-matched entry for the route. Because interface eno3 is the top-most best-matched entry in both Figure 39 and Figure 40, the operating system will use interface eno3 to route dNFS data traffic to the target. However, as shown in Figure 38, interface enp132s0f0 should be the interface used for that traffic.
To mitigate this traffic misdirection, a static route must be added to the operating system routing table. The static route forces the operating system to send dNFS data traffic between the IP addresses specified in the static route. In this case, the static route is:
To add a static route between interface enp132s0f0 and Unity link aggregate Ocp 0 0, run the ip route add or nmcli connection modify Linux command from Linux user root. Specify interface enp132s0f0 and parts of the network configuration (IP address, CIDR, and gateway) of Unity’s link aggregate Ocp 0 0:
ip route add mmm.mmm.mmm.mmm dev enp132s0f0
or
nmcli connection modify enp132s0f0 +ipv4.routes \ "mmm.mmm.mmm.mmm/cc ggg.ggg.ggg.ggg"
To verify that the static route appears in the routing table, run either:
command and filter for enp132s0f0. See Figure 41.
Because the static route defines a specific destination address (mmm.mmm.mmm.mmm) rather than the subnetwork IP address, the operating system considers the specific destination address as best-matched and uses it. The static route forces Ethernet traffic from interface enp132s0f0 (xxx.xxx.xxx.xxx) to Unity (mmm.mmm.mmm.mmm), as shown in Figure 42.
Manual changes to the routing table are not persistent across server reboots. If static routes should be persistent, consider creating routing configuration files. For more information, see the section Persistent static routes.
This example illustrates how static routing changes the paths taken on multiple interfaces configured with IP addresses from the same subnet with default routing (See Figure 43). NFS client Ethernet interfaces configured are:
Figure 43 shows that dNFS data traffic flows through interface eno3 rather than intended interfaces enp131s0f0 and enp3s0f0.
The following two figures show the corresponding routing table for Figure 43.
Figure 45 shows interface eno3 having the same subnet and CIDR as interfaces enp131s0f0, en3s0f0, and bond1. Since interface eno3 is the top-most best-matched entry in the routing table, all dNFS traffic flows through eno3.
To mitigate this traffic misdirection, static routes must be added to the operating system routing table. The static route forces the operating system to send dNFS traffic between the IP addresses specified in the static route. In this case, the static routes are:
To direct Ethernet traffic to flow through the three routes listed above, the following static routes are needed:
ip route add mmm.mmm.mmm.mmm dev bond1
ip route add ddd.ddd.ddd.ddd dev enp131s0f0
ip route add fff.fff.fff.fff dev enp3s0f0
Once the static routes are defined, the operating system chooses them because they best fit the destination and subnet mask/CIDR of the destination interface (see Figure 46).
Manual changes to the routing table are not persistent across server reboots. If static routes should be persistent, consider creating routing configuration files. For more information, see the section Persistent static routes.
To verify that the static routes appear in the routing table, run either:
and filter on the interface name:
To ensure that static routes persist across server reboots, create a static route configuration file in directory /etc/sysconfig/network-scripts for each of the static routes. Name the static route configuration file route-<NAS client interface>, substituting <NAS client interface> with the name of the network interface.
The following static route configuration files define the routes shown in Figure 50:
[root@r730xd-1 network-scripts]# cat route-bond1
ADDRESS0=mmm.mmm.mmm.mmm
NETMASK0=255.255.255.255
GATEWAY0=ggg.ggg.ggg.ggg
[root@r730xd-1 network-scripts]#
[root@r730xd-1 network-scripts]# cat route-enp131s0f0
ADDRESS0=ddd.ddd.ddd.ddd
NETMASK0=255.255.255.255
GATEWAY0=ggg.ggg.ggg.ggg
[root@r730xd-1 network-scripts]#
[root@r730xd-1 network-scripts]# cat route-enp3s0f0
ADDRESS0=fff.fff.fff.fff
NETMASK0=255.255.255.255
GATEWAY0=ggg.ggg.ggg.ggg
[root@r730xd-1 network-scripts]#