Deleting a file in Linux removes the file pointer that pointed to the old data, but it does not remove any of the old data from the storage array. So even though Linux views the space occupied by the delete data as free and available to use, the storage array does not. The storage array will show the deleted data as being consumed and does not release the space back to the pool of available storage on the storage array. For the storage array to release the space of deleted data back to the storage pool, several conditions must be met:
- Both the Linux operating system and storage array must support SCSI trim/unmap,
- The storage array must be able to create thinly provisioned volumes using SSDs,
- The Linux operating system must pass a SCSI trim/unmap command the storage array, and
- If a third-party logical volume manager is used, the volume manager must also support SCSI trim/unmap. The fourth option does not apply to Elastic Stack in the paper as Elastic Stack resides on volumes that are formatted with a file system and managed by Linux.
Being able to perform space reclamation is a major benefit with thinly provisioned volumes. Space reclamation helps reduce total cost of ownership by being able to reuse space as opposed to procuring additional storage arrays or scaling up an existing storage array.
The remainder of this section discusses the top three conditions listed above. The fourth condition will not be discussed as it does not apply to Elastic Stack in this paper.
With Oracle Linux 8.5 and PowerStore, the top two conditions are met. Both Oracle Linux 8.5 and PowerStore support SCSI trim/unmap, and all PowerStore volumes are thinly provisioned.
The third condition requires a little more explanation than a sentence or two.
With SSDs, data can only be written to new or erased cells on the SSD. If there is not enough available free space to support the file write, unused cells that once contained data must first be erased. Performing an erase when the file is being written can affect performance negatively. However, if data were erased from cells before data were attempted to be written, performance could improve. The SCSI trim command allows this to happen.
Linux uses a discard operation command to inform PowerStore which data are no longer being used so that the Storage array can erase the data from the SSD cells. Separating the erase from when the file is being written increases the performance of future writes as the erase operation is separated from the data write operation.
If there is a need to verify if the block device supports SCSI trim/unmap, view the value in /sys/block/<device>/queue/discard_max_bytes. A nonzero value indicates the block device supports SCSI trim/unmap.
[root@r730xd-2 ~]# cat /sys/block/sdh/queue/discard_max_bytes
268435456
[root@r730xd-2 ~]#
lsblk -D can also be used to determine if a particular block device or multipath block device supports the trim/unmap operation. If a device supports trim/unmap, lsblk will report nonzero values in columns DISC-GRAN and DISC-MAX. The below snippet reports device sde and multipath device mpatha as supporting SCSI trim/unmap.
[root@r730xd-2 ~]# lsblk -D
NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sdc 0 0B 0
sde 0 512K 256M 0
└─mpatha 0 512K 256M 0
In Linux 8, both ext4 and XFS support discard operations. Discard operations can be run using several different methods:
- Online discard
- Batch discard
- Periodic discard
The method chosen will depend on infrastructure requirements, but the general recommendation is to use either batch or periodic discards and to only use the online discard option if one of the following is true:
- The online discard operation is necessary to maintain performance
- The system’s workload is not conducive to the batch discard operation
To perform a batch or periodic discard operation in Linux, the fstrim command must be run against the device supporting the mounted ext4 or xfs file system. Using a batch or periodic discard operation can extend the life of the SSD.
Online discard
Online discard operations are specified when a file system is mounted. Discard operations run in real time without user interaction and only discard blocks that are transitioning from used to free. To enable online discard, specify the discard option (-o discard) when mounting a file system:
mount -t xfs -o discard /dev/mapper/esdata1 /opt/elasticsearch/data
mount -t xfs -o discard /dev/mapper/eslog1 /opt/elasticsearch/log
The discard option can also reside in /etc/fstab.
# cat /etc/fstab
UUID=13725bd7-d909-4f49-8854-433f84aa6870 /opt/elasticsearch/data xfs defaults 0 0
UUID=bc63eef2-99a0-4afe-8dd5-6e67bcc24209 /opt/elasticsearch/log xfs defaults 0 0
The UUID of a block device that is used to support a mounted file system can be determined by performing either of the following:
lsblk -o 'NAME,UUID'
blkid | grep <device>
# blkid | grep esdata1
/dev/mapper/esdata1: UUID="13725bd7-d909-4f49-8854-433f84aa6870" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="primary" PARTUUID="40fe0319-3ce3-45c6-a84b-e3e9070e1d40"
#
Batch discard
A batch discard operation discards all unused blocks in a mounted file system. If the file system is not mounted, an error will be reported. To perform a batch discard operation (SCSI trim), perform the following as a privileged user:
fstrim <file system> -v
# fstrim /opt/elasticsearch/log -v
/opt/elasticsearch/log: 6 GiB (6429634560 bytes) trimmed
When fstrim runs, it scans the mounted file system and sends SCSI unmap commands for each unused block it finds to PowerStore. PowerStore returns the unused blocks of storage to the storage pool.
If batch discard is used, it is recommended to create a cron schedule so the process is executed routinely:
# crontab -l
0 0 * * * /usr/bin/trim
For a list of fstrim arguments, see the Linux man page for fstrim.
Periodic discard
Periodic discard operations are batch discard operations performed by a system service called fstrim. By default, the service is loaded in the Linux 8 kernel, but it is not enabled, nor is it running.
[root@r730xd-2 ~]# systemctl status fstrim
● fstrim.service - Discard unused blocks
Loaded: loaded (/usr/lib/systemd/system/fstrim.service; static; vendor preset: disabled)
Active: inactive (dead)
[root@r730xd-2 ~]#
The corresponding fstrim scheduler service, fstrim.timer, is also loaded in the Linux 8 kernel. It too is not enabled nor is it running:
[root@r730xd-2 ~]# systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; disabled; vendor preset: disabled)
Active: inactive (dead)
Trigger: n/a
Docs: man:fstrim
[root@r730xd-2 ~]#
fstrim.timer triggers a fstrim operation weekly, every Monday at 00:00:00. The schedule can be verified by viewing the fstrim.timer definition with either command:
- cat /usr/lib/systemd/system/fstrim.timer
- systemctl cat fstrim.time
[root@r730xd-2 ~]# systemctl cat fstrim.timer
# /usr/lib/systemd/system/fstrim.timer
[Unit]
Description=Discard unused blocks once a week
Documentation=man:fstrim
[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
[Install]
WantedBy=timers.target
[root@r730xd-2 ~]#
To enable the fstrim timer and verify that systemd knows about the timer schedule, perform:
[root@r730xd-2 ~]# systemctl enable fstrim.timer
Created symlink /etc/systemd/system/timers.target.wants/fstrim.timer → /usr/lib/systemd/system/fstrim.timer.
[root@r730xd-2 ~]# systemctl list-timers --all
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2022-05-10 12:20:11 CDT 19min left Tue 2022-05-10 10:50:39 CDT 1h 9min ago dnf-makecache.timer dnf-makecache.service
Wed 2022-05-11 00:00:00 CDT 11h left Tue 2022-05-10 00:00:39 CDT 11h ago mlocate-updatedb.timer mlocate-updatedb.service
Wed 2022-05-11 00:00:00 CDT 11h left Tue 2022-05-10 00:00:39 CDT 11h ago unbound-anchor.timer unbound-anchor.service
Wed 2022-05-11 06:03:39 CDT 18h left Tue 2022-05-10 06:03:39 CDT 5h 56min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
n/a n/a n/a n/a fstrim.timer fstrim.service
5 timers listed.
[root@r730xd-2 ~]#
To start the fstrim.timer schedule, perform:
[root@r730xd-2 ~]# systemctl start fstrim.timer
[root@r730xd-2 ~]# systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: disabled)
Active: active (waiting) since Tue 2022-05-10 12:01:10 CDT; 6s ago
Trigger: Mon 2022-05-16 00:00:00 CDT; 5 days left
Docs: man:fstrim
May 10 12:01:10 r730xd-2.techsol.local systemd[1]: Started Discard unused blocks once a week.
[root@r730xd-2 ~]# systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2022-05-10 12:20:11 CDT 18min left Tue 2022-05-10 10:50:39 CDT 1h 11min ago dnf-makecache.timer dnf-makecache.service
Wed 2022-05-11 00:00:00 CDT 11h left Tue 2022-05-10 00:00:39 CDT 12h ago mlocate-updatedb.timer mlocate-updatedb.service
Wed 2022-05-11 00:00:00 CDT 11h left Tue 2022-05-10 00:00:39 CDT 12h ago unbound-anchor.timer unbound-anchor.service
Wed 2022-05-11 06:03:39 CDT 18h left Tue 2022-05-10 06:03:39 CDT 5h 58min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2022-05-16 00:00:00 CDT 5 days left n/a n/a fstrim.timer fstrim.service
5 timers listed.
Pass --all to see loaded but inactive timers, too.
[root@r730xd-2 ~]#
