Home > Storage > PowerStore > Databases and Data Analytics > Dell PowerStore: Oracle Best Practices > Shared subnets
If multiple network paths are needed for dNFS/kNFS control traffic and dNFS data traffic, Oracle recommends that each path uses a separate subnet. However, using separate subnets may not always be possible.
This section discusses the use of subnet sharing on network paths for Oracle dNFS.
When configuring dNFS to use a subnet that is shared with another network interface, additional configuration, including IPv4 network routing filter and static routing, is required.
According to Oracle, if any IP address used by dNFS is from a subnet on another NIC interface, the following additional configuration changes must be made:
Note: For more information about dontroute and oranfstab, see Oracle Database—Database Installation Guide.
For more information about shared subnets, see Static routing.
Linux 8 follows the recommendations of ingress filtering for multihomed networks (http://tools.ietf.org/html/rfc3704). These routing filters must be relaxed for multiple NIC interfaces in the same server to use the same subnet. Before configuring network interfaces to use the same subnet, ensure that you relax routing filters.
If the Oracle 21c preinstall rpm is used to configure the operating system before installing Oracle, the routing filters will be relaxed appropriately.
With OL 8u8, rp_filter is set to 2 for all network interfaces including the default interface. A value of 2 (loose filtering) is acceptable. Should no filtering (rp_filter = 0) or strict filtering (rp_filter = 1) be needed, see Oracle Grid Infrastructure—Grid Infrastructure Installation and Upgrade Guide, and My Oracle Support Doc ID 1286796.1.
If dNFS multipathing is needed and there is only one subnet, loose filtering (rp_filter = 2) needs to be set.
Setting rp_filter to an incorrect value can cause interconnect packets to be blocked or discarded in a RAC configuration.
rp_filter value | Description |
0 | No filtering |
1 | Strict filtering |
2 | Loose filtering |
To verify that IPv4 routing filters have been relaxed, look at the rp_filter settings in /etc/sysctl.conf. The values should be 2.
If IPv4 reverse path filtering is not set, add or change the IPv4 rp_filters in /etc/sysctl.conf and make them active.
For more information about channel bonded interfaces for NFS control traffic, see Oracle MOS notes:
Starting with OL kernel 2.6.31, Oracle RAC systems that use multiple NICs for the private interconnect must have parameter rp_filter set appropriately.
When dNFS traffic on the NFS client must route traffic to an IP address from a subnet already in use on the NFS client, static routing must be configured. A static route must be defined for each of the network addresses from a shared subnet and used by dNFS as a route. This requirement includes defining a static route for dNFS data traffic should it use the same subnet as dNFS control traffic. If static routes are not defined, automatic load balancing and performance tuning of dNFS will not operate as expected per the dNFS path definitions in file oranfstab.
The remainder of this section provides two examples of static routing. The first example considers two interfaces sharing a subnet, and the second example considers four network interfaces sharing a subnet.
The following figure illustrates a scenario where an NFS client has two interfaces. One interface, enp132s0f0, is intended to be dedicated to dNFS traffic, and the other, eno3, for general public Ethernet traffic. Both interfaces use a network IP address from the same subnet, and default routing is defined. The figure shows dNFS traffic flowing through interface eno3 rather than the intended interface enp132s0f0.
The following two figures show the corresponding routing table for Figure 58.
If default routing and IP addresses from the same subnet are used, the operating system searches the routing table for the route that best matches the destination address and mask. The operating system will use the best-matched route found. Because interfaces eno3 and enp132s0f0 share the same subnet (nnn.nnn.nnn.nnn) and CIDR (cc) [or Destination nnn.nnn.nnn.nnn and subnet mask (Genmask sss.sss.sss.sss)], the operating system considers both entries as best-matched. When multiple best-matched entries exist, the operating system selects the top-most best-matched entry for the route. Because interface eno3 is the top-most best-matched entry in both Figure 59 and Figure 60, the operating system will use interface eno3 to route dNFS data traffic to the target. However, as shown in Figure 58, interface enp132s0f0 should be the interface used for that traffic.
To mitigate this traffic misdirection, a static route must be added to the operating system routing table. The static route forces the operating system to send dNFS data traffic between the IP addresses specified in the static route. In this case, the static route is:
To add a static route between interface enp132s0f0 and PowerStore BaseEnclosure-bond0, run the ip route add command. Specify the interface name enp132s0f0 and the IP address of PowerStore BaseEnclosure-bond0:
ip route add mmm.mmm.mmm.mmm dev enp132s0f0
or
nmcli connection modify enp132s0f0 +ipv4.routes "mmm.mmm.mmm.mmm/cc ggg.ggg.ggg.ggg"
To verify that the static route appears in the routing table, run either a netstat -r,
ip route, or nmcli connection show command and filter for enp132s0f0:
Because the static route defines a specific destination IP address rather than the subnetwork IP address, the operating system considers the specific destination IP address as best-matched and uses it. The static route forces Ethernet traffic from interface enp132s0f0 (xxx.xxx.xxx.xxx) to PowerStore (mmm.mmm.mmm.mmm), as shown in the following figure:
Manual changes to the routing table are not persistent across server reboots. If static routes should be persistent, consider creating routing configuration files. For more information, see the section Persistent static routes.
This example illustrates how static routing changes the paths taken on multiple interfaces configured with IP addresses from the same subnet.
Figure 63 illustrates a scenario where an NFS client has four configured interfaces.
All four interfaces use a different network IP address from the same subnet, and default routing is defined. The figure shows that dNFS data traffic would flow through interface eno3 rather than intended interfaces enp131s0f0 and enp3s0f0.
The following two figures show the corresponding routing table for Figure 63.
All dNFS traffic would again flow through interface eno3 because it is the top-most best-matched entry in the routing table that uses the same subnet and CIDR.
To mitigate this traffic misdirection, static routes must be added to the operating system routing table. The static route forces the operating system to send dNFS data traffic between the IP addresses specified in the static route. In this case, the static routes are:
To direct Ethernet traffic to flow through all the intended interfaces (enp131s0f0, enp3s0f0, and bond1), the following static routes are needed:
ip route add mmm.mmm.mmm.mmm dev bond1
ip route add ddd.ddd.ddd.ddd dev enp131s0f0
ip route add fff.fff.fff.fff dev enp3s0f0
Once the static routes are defined, the operating system chooses static routes because they best fit the destination and subnet mask/CIDR of the destination interface (see Figure 66).
Manual changes to the routing table are not persistent across server reboots. If static routes should be persistent, consider creating routing configuration files. For more information, see the section Persistent static routes.
To verify that the static route appears in the routing table, run either netstat -r,
ip route, or nmcli connection show command and filter on the interface name:
To ensure that static routes persist across server reboots, create a static route configuration file in directory /etc/sysconfig/network-scripts for each of the static routes. Name the static route configuration file route-<NAS client interface>.
The following static route configuration files define the routes shown in Figure 70:
[root@r730xd-1 network-scripts]# cat route-bond1
ADDRESS0=mmm.mmm.mmm.mmm
NETMASK0=255.255.255.255
GATEWAY0=ggg.ggg.ggg.ggg
[root@r730xd-1 network-scripts]#
[root@r730xd-1 network-scripts]# cat route-enp131s0f0
ADDRESS0=ddd.ddd.ddd.ddd
NETMASK0=255.255.255.255
GATEWAY0=ggg.ggg.ggg.ggg
[root@r730xd-1 network-scripts]#
[root@r730xd-1 network-scripts]# cat route-enp3s0f0
ADDRESS0=fff.fff.fff.fff
NETMASK0=255.255.255.255
GATEWAY0=ggg.ggg.ggg.ggg
[root@r730xd-1 network-scripts]#