OneFS Time Synchronization and NTP
Fri, 11 Mar 2022 16:08:05 -0000
|Read Time: 0 minutes
OneFS provides a network time protocol (NTP) service to ensure that all nodes in a cluster can easily be synchronized to the same time source. This service automatically adjusts a cluster’s date and time settings to that of one or more external NTP servers.
You can perform NTP configuration on a cluster using the isi ntp command line (CLI) utility, rather than modifying the nodes’ /etc/ntp.conf files manually. The syntax for this command is divided into two parts: servers and settings. For example:
# isi ntp settings Description: View and modify cluster NTP configuration. Required Privileges: ISI_PRIV_NTP Usage: isi ntp settings <action> [--timeout <integer>] [{--help | -h}] Actions: modify Modify cluster NTP configuration. view View cluster NTP configuration. Options: Display Options: --timeout <integer> Number of seconds for a command timeout (specified as 'isi --timeout NNN <command>'). --help | -h Display help for this command.
There is also an isi_ntp_config CLI command available in OneFS that provides a richer configuration set and combines the server and settings functionality:
Usage: isi_ntp_config COMMAND [ARGUMENTS ...] Commands: help Print this help and exit. list List all configured info. add server SERVER [OPTION] Add SERVER to ntp.conf. If ntp.conf is already configured for SERVER, the configuration will be replaced. You can specify any server option. See NTP.CONF(5) delete server SERVER Remove server configuration for SERVER if it exists. add exclude NODE [NODE...] Add NODE (or space separated nodes) to NTP excluded entry. Excluded nodes are not used for NTP communication with external NTP servers. delete exclude NODE [NODE...] Delete NODE (or space separated Nodes) from NTP excluded entry. keyfile KEYFILE_PATH Specify keyfile path for NTP auth. Specify "" to clear value. KEYFILE_PATH has to be a path under /ifs. chimers [COUNT | "default"] Display or modify the number of chimers NTP uses. Specify "default" to clear the value.
By default, if the cluster has more than three nodes, three of the nodes are selected as chimers. Chimers are nodes which can contact the external NTP servers. If the cluster consists of three nodes or less, only one node is selected as a chimer. If no external NTP server is set, they use the local clock instead. The other non-chimer nodes use the chimer nodes as their NTP servers. The chimer nodes are selected by the lowest node number which is not excluded from chimer duty.
If a node is configured as a chimer. its /etc/ntp.conf entry will resemble: # This node is one of the 3 chimer nodes that can contact external NTP # servers. The non-chimer nodes will use this node as well as the other # chimers as their NTP servers. server time.isilon.com # The other chimer nodes on this cluster: server 192.168.10.150 iburst server 192.168.10.151 iburst # If none or bad connection to external servers this node may become # the time server for this cluster. The system clock will be a time # source and run at a high stratum
Besides managing NTP servers and authentication, you can exclude individual nodes from communicating with external NTP servers.
The local clock of the node is set as an NTP server at a high stratum level. In NTP, a server with lower stratum number is preferred, so if an external NTP server is set, the system prefers an external time server if configured. The stratum level for the chimer is determined by the chimer number. The first chimer is set to stratum 9, the second to stratum 11, and the others continue to increment the stratum number by 2. This is so the non-chimer nodes prefer to get the time from the first chimer if available.
For a non-chimer node, its /etc/ntp.conf entry will resemble:
# This node is _not_ one of the 3 chimer nodes that can contact external # NTP servers. These are the cluster's chimer nodes: server 192.168.10.149 iburst true server 192.168.10.150 iburst true server 192.168.10.151 iburst true
When configuring NTP on a cluster, you can specify more than one NTP server to synchronize the system time from. This ability allows for full redundancy of ysnc targets. The cluster periodically contacts the server or servers and adjusts the time, date or both as necessary, based on the information it receives.
You can use the isi_ntp_config CLI command to configure which NTP servers a cluster will reference. For example, the following syntax adds the server time.isilon.com:
# isi_ntp_config add server time.isilon.com
Alternatively, you can manage the NTP configuration from the WebUI by going to Cluster Management > General Settings > NTP.
NTP also provides basic authentication-based security using symmetrical keys, if preferred.
If no NTP servers are available, Windows Active Directory (AD) can synchronize domain members to a primary clock running on the domain controller or controllers. If there are no external NTP servers configured and the cluster is joined to AD, OneFS uses the Windows domain controller as the NTP time server. If the cluster and domain time become out of sync by more than four minutes, OneFS generates an event notification.
Be aware that if the cluster and Active Directory drift out of time sync by more than five minutes, AD authentication will cease to function.
If both NTP server and domain controller are not available, you can manually set the cluster’s time, date and time zone using the isi config CLI command. For example:
1. Run the isi config command. The command-line prompt changes to indicate that you are in the isi config subsystem:
# isi config Welcome to the Isilon IQ configuration console. Copyright (c) 2001-2017 EMC Corporation. All Rights Reserved. Enter 'help' to see list of available commands. Enter 'help <command>' to see help for a specific command. Enter 'quit' at any prompt to discard changes and exit. Node build: Isilon OneFS v8.2.2 B_8_2_2(RELEASE)Node serial number: JWXER170300301 >>>
2. Specify the current date and time by running the date command. For example, the following command sets the cluster time to 9:20 AM on April 23, 2020:
>>> date 2020/04/23 09:20:00 Date is set to 2020/04/23 09:20:00
3. The help timezone command lists the available timezones. For example:
>>> help timezone timezone [<timezone identifier>] Sets the time zone on the cluster to the specified time zone. Valid time zone identifiers are: Greenwich Mean Time Eastern Time Zone Central Time Zone Mountain Time Zone Pacific Time Zone Arizona Alaska Hawaii Japan Advanced
4. To verify the currently configured time zone, run the timezone command. For example:
>>> timezone The current time zone is: Greenwich Mean Time
5. To change the time zone, enter the timezone command followed by one of the displayed options. For example, the following command changes the time zone to Alaska:
>>> timezone Alaska Time zone is set to Alaska
A message confirming the new time zone setting displays. If your preferred time zone did not display when you ran the help timezone command, enter timezone Advanced. After a warning screen displays, you will see a list of regions. When you select a region, a list of specific time zones for that region appears. Select the preferred time zone (you may need to scroll), and enter OK or Cancel until you return to the isi config prompt.
6. When done, run the commit command to save your changes and exit isi config.
>>> commit Commit succeeded.
Alternatively, you can manage these time and date parameters from the WebUI by going to Cluster Management > General Settings > Date and Time.
Author: Nick Trimbee