This section describes the vSAN storage design for the five types of Splunk Enterprise deployment on a VxRail cluster described earlier.
In this solution, we used a default vSAN storage policy and a custom policy:
- The default vSAN policy is used for the home files and the operating system Virtual Machine Disk (VMDK) files of all virtual machines.
- A newly created policy for the disk drive keeps Splunk data on the Splunk indexer virtual machines. The number of disk stripes per object is set to 10 which can drive higher levels of I/O parallelism if, and only if, the capacity devices are a significant source of contention.
- A vSAN storage policy with failures to tolerate (FTT) of 1 and failure tolerance method of RAID 1 (mirroring) creates a full copy of the data, which requires twice the capacity of the workload.
The following table shows the vSAN storage policies that are defined for the virtual machines:
Table 7. vSAN storage policy configuration
vSAN Default Storage Policy | - Number of failures to tolerate = 1
- Failure tolerance method = RAID 1
- Disable object checksum = No
- Force provisioning = No
- IOPS limit for object = 0
- Number of disk stripes per object = 1
- Object space reservation (%) = 0
| This policy is the vSAN default policy. It is used for the VM home files and operating system VMDK files of virtual machines created in this solution. |
User created Splunk-Data-Policy | - Number of failures to tolerate = 1
- Failure tolerance method = RAID 1
- Disable object checksum = No
- Force provisioning = No
- IOPS limit for object = 0
- Number of disk stripes per object = 10
- Object space reservation (%) = 0
| This policy is used for Splunk indexer data storage. |
Table 8. vSAN storage design for Splunk
Single Instance 50 GB/day with 90-day retention | Single instance combined search head and indexer | 1 | 300 GB | 3 TB |
Multi-instance 500 GB/day with 90-day retention | Search Head | 1 | 300 GB | 0 |
Indexer | 2 | 300 GB | 13.9 TB |
Admin Server | 1 | 150 GB | 0 |
Multi-instance 1000 GB/day with 90-day retention | Search Head | 1 | 300 GB | 0 |
Indexer | 5 | 300 GB | 10.8 TB |
Admin Server | 1 | 150 GB | 0 |
Multi-instance 1000 GB/day with > 90-day retention | Search Head | 1 | 300 GB | 0 |
Indexer | 5 | 300 GB | 2.1 TB |
Admin Server | 1 | 150 GB | 0 |
Multi-instance 1000 GB/day with > 90-day retention and Indexer High Availability | Search Head | 1 | 300 GB | 0 |
Indexer | 5 | 300 GB | 2.1 TB |
Admin Server | 1 | 150 GB | 0 |
In the solution shown in Figure 2, a four-node Isilon H500 cluster is used for the Splunk deployment to provide configurable retention for cold buckets. The following tables show the detailed Isilon node configuration and Isilon storage design for Splunk:
Table 9. Isilon node configuration
Two Intel Xeon Processor E5-2630 v4 | 10 cores | 128 GB | 1.6 TB or 3.2 TB (1 or 2 drives) | 120 – 960 TB | 2 x 10 GbE 2 x 25 GbE 2 x 40 GbE |
Table 10. Isilon storage design for Splunk
Multi-instance 1000 GB/day with > 90-day Retention | Indexer | 5 | 10.8 TB |
Multi-instance 1000 GB/day with > 90-day Retention and Indexer High Availability | Indexer | 5 | 10.8 TB |
For the Isilon configuration, we followed these best practices:
- Enabled SmartPools settings across all four Isilon nodes and used an SSD as L3 cache for metadata read acceleration
- Enabled SmartConnect to provide automatic client connection load balancing and failover capabilities
- Enabled SmartCache for write performance
- Optimized for concurrent access for data access pattern
- Used 10 Gb/s external network for data connection (25 Gb/s is applicable)
- Increased network MTU to 9000 (Jumbo Frames)
Splunk and Dell Technologies recommend that NFS storage, including PowerScale storage, is only used for cold and frozen data, never for hot/warm. For details about system requirements, see the Splunk Enterprise Installation Manual.
The Splunk storage sizing tool at http://splunk-sizing.appspot.com/ can provide an estimated storage requirement for beginning your design planning. However, Splunk states that this tool is not 100 percent accurate. A Dell technologist can assist you with the complete solution design.
