OneFS Account Security Policy
Thu, 20 Jul 2023 16:23:21 -0000
|Read Time: 0 minutes
Another of the core security enhancements introduced in OneFS 9.5 is the ability to enforce strict user account security policies. This is required for compliance with both private and public sector security mandates. For example, the account policy restriction requirements expressed within the U.S. military STIG requirements stipulate:
Requirement | Description |
---|---|
Delay | The OS must enforce a delay of at least 4 seconds between logon prompts following a failed logon attempt. |
Disable | The OS must disable account identifiers (individuals, groups, roles, and devices) after 35 days of inactivity. |
Limit | The OS must limit the number of concurrent sessions to ten for all accounts and/or account types. |
To directly address these security edicts, OneFS 9.5 adds the following account policy restriction controls:
Account policy function | Details |
---|---|
Delay after failed login |
|
Disable inactive accounts |
|
Concurrent session limit |
|
Architecture
OneFS provides a variety of access mechanisms for administering a cluster. These include SSH, serial console, WebUI, and platform API, all of which use different underlying access methods. The serial console and SSH are standard FreeBSD third-party applications and are accounted for per node, whereas the WebUI and pAPI use HTTP module extensions to facilitate access to the system and services and are accounted for cluster-wide. Before OneFS 9.5, there was no common mechanism to represent or account for sessions across these disparate applications.
Under the hood, the OneFS account security policy framework encompasses the following high-level architecture:
With SSH, there’s no explicit or reliable “log-off” event sent to OneFS, beyond actually disconnecting the connection. As such, accounting for active sessions can be problematic and unreliable, especially when connections time out or unexpectedly disconnect. However, OneFS does include an accounting database that stores records of system activities like user login and logout, which can be queried to determine active SSH sessions. Each active SSH connection has an isi_ssh_d process owned by the account associated with it, and this information can be gathered via standard syscalls. OneFS enumerates the number of SSHD processes per account to calculate the total number of active established sessions. This value is then used as part of the total concurrent administrative sessions limit. Since SSH only supports user access through the system zone, there is no need for any zone-aware accounting.
The WebUI and platform API use JSON web tokens (JWTs) for authenticated sessions. OneFS stores the JWTs in the cluster-wide kvstore, and access policy uses valid session tokens in the kvstore to account for active sessions when a user logs on through the WebUI or pAPI. When the user logs off, the associated token is removed, and a message is sent to JWT service with an explicit log off notification. If a session times out or disconnects, the JWT service will not get an event, but the tokens have a limited, short lifespan, and any expired tokens are purged from the list on a scheduled basis in conjunction with the JWT timer. OneFS enumerates the unique session IDs associated with each user’s JWT tokens in the kvstore to get a number of active WebUI and pAPI sessions to use as part of user’s session limit check.
For serial console access accounting, the process table will have information when an STTY connection is active, and OneFS extrapolates user data from it to determine the session count, similar to ssh with a syscall for process data. There is an accounting database that stores records of system activities like user login and logout, which is also queried for active console sessions. Serial console access is only from the system zone, so there is no need for zone-aware accounting.
An API call retrieves user session data from the process table and kvstore to calculate number of user active sessions. As such, the checking and enforcement of session limits is performed in similar manner to the verification of user privileges for SSH, serial console, or WebUI access.
Delaying failed login reconnections
OneFS 9.5 provides the ability to enforce a configurable delay period. This delay is specified in seconds, after which every unsuccessful authentication attempt results in the user being denied the ability to reconnect to the cluster until after the configured delay period has passed. The login delay period is defined in seconds through the FailedLoginDelayTime global attribute and, by default, OneFS is configured for no delay through a FailedLoginDelayTime value of 0. When a cluster is placed into hardened mode with the STIG policy enacted, the delay value is automatically set to 4 seconds. Note that the delay happens in the lsass client, so that the authentication service is not affected.
The configured failed login delay time limit can be viewed with following CLI command:
# isi auth settings global view Send NTLMv2: No Space Replacement: Workgroup: WORKGROUP Provider Hostname Lookup: disabled Alloc Retries: 5 User Object Cache Size: 47.68M On Disk Identity: native RPC Block Time: Now RPC Max Requests: 64 RPC Timeout: 30s Default LDAP TLS Revocation Check Level: none System GID Threshold: 80 System UID Threshold: 80 Min Mapped Rid: 2147483648 Group UID: 4294967292 Null GID: 4294967293 Null UID: 4294967293 Unknown GID: 4294967294 Unknown UID: 4294967294 Failed Login Delay Time: Now Concurrent Session Limit: 0
Similarly, the following syntax will configure the failed login delay time to a value of 4 seconds:
# isi auth settings global modify --failed-login-delay-time 4s # isi auth settings global view | grep -i delay Failed Login Delay Time: 4s
However, when a cluster is put into STIG hardening mode, the “Concurrent sessions limit” is automatically set to 10.
# isi auth settings global view | grep -i delay Failed Login Delay Time: 10s
The delay time after login failure can also be configured from the WebUI under Access > Settings > Global provider settings:
The valid range of the FailedLoginDelayTime global attribute is from 0 to 65535, and the delay time is limited to the same cluster node.
Note that this maximum session limit is only applicable to administrative logins.
Disabling inactive accounts
In OneFS 9.5, any user account that has been inactive for a configurable duration can be automatically disabled. Administrative intervention is required to re-enable a deactivated user account. The last activity time of a user is determined by their previous logon, and a timer runs every midnight during which all “inactive” accounts are disabled. If the last logon record for a user is unavailable, or stale, the timestamp when the account was enabled is taken as their last activity instead. If inactivity tracking is enabled after the last logon (or enabled) time of a user, the inactivity tracking time is considered for inactivity period.
This feature is disabled by default in OneFS, and all users are exempted from inactivity tracking until configured otherwise. However, individual accounts can be exempted from this behavior, and this can be configured through the user-specific DisableWhenInactive attribute. For example:
# isi auth user view user1 | grep -i inactive Disable When Inactive: Yes # isi auth user modify user1 --disable-when-inactive 0 # isi auth user view user1 | grep -i inactive Disable When Inactive: No
If a cluster is put into STIG hardened mode, the value for the MaxInactivityDays parameter is automatically reconfigured to 35, meaning a user will be disabled after 35 days of inactivity. All the local users are removed from exemption when in STIG hardened mode.
Note that this functionality is limited to only the local provider and does not apply to file providers.
The inactive account disabling configuration can be viewed from the CLI with the following syntax. In this example, the MaxInactivityDays attribute is configured for 35 days:
# isi auth local view system Name: System Status: active Authentication: Yes Create Home Directory: Yes Home Directory Template: /ifs/home/%U Lockout Duration: Now Lockout Threshold: 0 Lockout Window: Now Login Shell: /bin/zsh Machine Name: Min Password Age: Now Max Password Age: 4W Min Password Length: 15 Password Prompt Time: 2W Password Complexity: - Password History Length: 0 Password Chars Changed: 8 Password Percent Changed: 50 Password Hash Type: NTHash Max Inactivity Days: 35
Inactive account disabling can also be configured from the WebUI under Access > Authentication providers > Local provider:
The valid range of the MaxInactivityDays parameter is from 0 to UINT_MAX. As such, the following CLI syntax will configure the maximum number of days a user account can be inactive before it will be disabled to 10 days:
# isi auth local modify system --max-inactivity-days 10 # isi auth local view system | grep -i inactiv Max Inactivity Days: 0tem –max-inactivity-days 10
Setting this value to 0 days will disable the feature:
# isi auth local modify system --max-inactivity-days 0 # isi auth local view system | grep -i inactiv Max Inactivity Days: 0tem –max-inactivity-days 0
Inactivity account disabling, as well as password expiry, can also be configured granularly, per user account. For example, user1 has a default configuration of the Disable When Inactive threshold set to No.
# isi auth users view user1 Name: user1 DN: CN=user1,CN=Users,DC=GLADOS DNS Domain: - Domain: GLADOS Provider: lsa-local-provider:System Sam Account Name: user1 UID: 2000 SID: S-1-5-21-1839173366-2940572996-2365153926-1000 Enabled: Yes Expired: No Expiry: - Locked: No Email: - GECOS: - Generated GID: No Generated UID: No Generated UPN: Yes Primary Group ID: GID:1800 Name: Isilon Users Home Directory: /ifs/home/user1 Max Password Age: 4W Password Expired: No Password Expiry: 2023-06-15T17:45:55 Password Last Set: 2023-05-18T17:45:55 Password Expired: No Last Logon: - Shell: /bin/zsh UPN: user1@GLADOS User Can Change Password: Yes Disable When Inactive: No
The following CLI command will activate the account inactivity disabling setting and enable password expiry for the user1 account:
# isi auth users modify user1 --disable-when-inactive Yes --password-expires Yes
Inactive account disabling can also be configured from the WebUI under Access > Membership and roles > Users > Providers:
Limiting concurrent sessions
OneFS 9.5 can limit the number of administrative sessions active on a OneFS cluster node, and all WebUI, SSH, pAPI, and serial console sessions are accounted for when calculating the session limit. The SSH and console session count is node-local, whereas WebUI and pAPI sessions are tracked cluster-wide. As such, the formula used to calculate a node’s total active sessions is as follows:
Total active user sessions on a node = Total WebUI and pAPI sessions across the cluster + Total SSH and Console sessions on the node
This feature leverages the cluster-wide session management through JWT for calculating the total number of sessions on a cluster’s node. By default, OneFS 9.5 has no configured limit, and the Concurrent Session Limit parameter has a value of 0. For example:
# isi auth settings global view Send NTLMv2: No Space Replacement: Workgroup: WORKGROUP Provider Hostname Lookup: disabled Alloc Retries: 5 User Object Cache Size: 47.68M On Disk Identity: native RPC Block Time: Now RPC Max Requests: 64 RPC Timeout: 30s Default LDAP TLS Revocation Check Level: none System GID Threshold: 80 System UID Threshold: 80 Min Mapped Rid: 2147483648 Group UID: 4294967292 Null GID: 4294967293 Null UID: 4294967293 Unknown GID: 4294967294 Unknown UID: 4294967294 Failed Login Delay Time: Now Concurrent Session Limit: 0
The following CLI syntax will configure Concurrent Session Limit to a value of 5:
# isi auth settings global modify –-concurrent-session-limit 5 # isi auth settings global view | grep -i concur Concurrent Session Limit: 5
Once the session limit has been exceeded, attempts to connect, in this case as root through SSH, will be met with the following Access denied error message:
login as: root Keyboard-interactive authentication prompts from server: | Password: End of keyboard-interactive prompts from server Access denied password:
The concurrent sessions limit can also be configured from the WebUI under Access > Settings > Global provider settings:
However, when a cluster is put into STIG hardening mode, the concurrent session limit is automatically set to a maximum of 10 sessions.
Note that this maximum session limit is only applicable to administrative logins.
Performance
Disabling an account after a period of inactivity in OneFS requires a SQLite database update every time a user has successfully logged on to the OneFS cluster. After a successful logon, the time to logon is recorded in the database, which is later used to compute the inactivity period.
Inactivity tracking is disabled by default in OneFS 9.5, but can be easily enabled by configuring the MaxInactivityDays attribute to a non-zero value. In cases where inactivity tracking is enabled and many users are not exempt from inactivity tracking, a significant number of logons within a short period of time can generate significant SQLite database requests. However, OneFS consolidates multiple database updates during user logon to a single commit to minimize the overall load.
Troubleshooting
When it comes to troubleshooting OneFS account security policy configurations, there are these main logfiles to check:
- /var/log/lsassd.log
- /var/log/messages
- /var/log/isi_papi_d.log
For additional reporting detail, debug level logging can be enabled on the lsassd.log file with the following CLI command:
# /usr/likewise/bin/lwsm set-log-level lsass – debug
When finished, logging can be returned to the regular error level:
# /us/likewise/bin/lwsm set-log-level lsass - error
Author: Nick Trimbee
Related Blog Posts
OneFS and HTTP Security
Mon, 22 Apr 2024 20:35:30 -0000
|Read Time: 0 minutes
To enable granular HTTP security configuration, OneFS provides an option to disable nonessential HTTP components selectively. This can help reduce the overall attack surface of your infrastructure. Disabling a specific component’s service still allows other essential services on the cluster to continue to run unimpeded. In OneFS 9.4 and later, you can disable the following nonessential HTTP services:
Service | Description |
PowerScaleUI | The OneFS WebUI configuration interface. |
Platform-API-External | External access to the OneFS platform API endpoints. |
Rest Access to Namespace (RAN) | REST-ful access by HTTP to a cluster’s /ifs namespace. |
RemoteService | Remote Support and In-Product Activation. |
SWIFT (deprecated) | Deprecated object access to the cluster using the SWIFT protocol. This has been replaced by the S3 protocol in OneFS. |
You can enable or disable each of these services independently, using the CLI or platform API, if you have a user account with the ISI_PRIV_HTTP RBAC privilege.
You can use the isi http services CLI command set to view and modify the nonessential HTTP services:
# isi http services list ID Enabled ------------------------------ Platform-API-External Yes PowerScaleUI Yes RAN Yes RemoteService Yes SWIFT No ------------------------------ Total: 5
For example, you can easily disable remote HTTP access to the OneFS /ifs namespace as follows:
# isi http services modify RAN --enabled=0
You are about to modify the service RAN. Are you sure? (yes/[no]): yes
Similarly, you can also use the WebUI to view and edit a subset of the HTTP configuration settings, by navigating to Protocols > HTTP settings:
That said, the implications and impact of disabling each of the services is as follows:
Service | Disabling impacts |
WebUI | The WebUI is completely disabled, and access attempts (default TCP port 8080) are denied with the warning Service Unavailable. Please contact Administrator. If the WebUI is re-enabled, the external platform API service (Platform-API-External) is also started if it is not running. Note that disabling the WebUI does not affect the PlatformAPI service. |
Platform API | External API requests to the cluster are denied, and the WebUI is disabled, because it uses the Platform-API-External service. Note that the Platform-API-Internal service is not impacted if/when the Platform-API-External is disabled, and internal pAPI services continue to function as expected. If the Platform-API-External service is re-enabled, the WebUI will remain inactive until the PowerScaleUI service is also enabled. |
RAN | If RAN is disabled, the WebUI components for File System Explorer and File Browser are also automatically disabled. From the WebUI, attempts to access the OneFS file system explorer (File System > File System Explorer) fail with the warning message Browse is disabled as RAN service is not running. Contact your administrator to enable the service. This same warning also appears when attempting to access any other WebUI components that require directory selection. |
RemoteService | If RemoteService is disabled, the WebUI components for Remote Support and In-Product Activation are disabled. In the WebUI, going to Cluster Management > General Settings and selecting the Remote Support tab displays the message The service required for the feature is disabled. Contact your administrator to enable the service. In the WebUI, going to Cluster Management > Licensing and scrolling to the License Activation section displays the message The service required for the feature is disabled. Contact your administrator to enable the service. |
SWIFT | Deprecated object protocol and disabled by default. |
You can use the CLI command isi http settings view to display the OneFS HTTP configuration:
# isi http settings view Access Control: No Basic Authentication: No WebHDFS Ran HTTPS Port: 8443 Dav: No Enable Access Log: Yes HTTPS: No Integrated Authentication: No Server Root: /ifs Service: disabled Service Timeout: 8m20s Inactive Timeout: 15m Session Max Age: 4H Httpd Controlpath Redirect: No
Similarly, you can manage and change the HTTP configuration using the isi http settings modify CLI command.
For example, to reduce the maximum session age from four to two hours:
# isi http settings view | grep -i age Session Max Age: 4H # isi http settings modify --session-max-age=2H # isi http settings view | grep -i age Session Max Age: 2H
The full set of configuration options for isi http settings includes:
Option | Description |
--access-control <boolean> | Enable Access Control Authentication for the HTTP service. Access Control Authentication requires at least one type of authentication to be enabled. |
--basic-authentication <boolean> | Enable Basic Authentication for the HTTP service. |
--webhdfs-ran-https-port <integer> | Configure Data Services Port for the HTTP service. |
--revert-webhdfs-ran-https-port | Set value to system default for --webhdfs-ran-https-port. |
--dav <boolean> | Comply with Class 1 and 2 of the DAV specification (RFC 2518) for the HTTP service. All DAV clients must go through a single node. DAV compliance is NOT met if you go through SmartConnect, or using 2 or more node IPs. |
--enable-access-log <boolean> | Enable writing to a log when the HTTP server is accessed for the HTTP service. |
--https <boolean> | Enable the HTTPS transport protocol for the HTTP service. |
--https <boolean> | Enable the HTTPS transport protocol for the HTTP service. |
--integrated-authentication <boolean> | Enable Integrated Authentication for the HTTP service. |
--server-root <path> | Document root directory for the HTTP service. Must be within /ifs. |
--service (enabled | disabled | redirect | disabled_basicfile) | Enable/disable the HTTP Service or redirect to WebUI or disabled BasicFileAccess. |
--service-timeout <duration> | The amount of time (in seconds) that the server will wait for certain events before failing a request. A value of 0 indicates that the service timeout value is the Apache default. |
--revert-service-timeout | Set value to system default for --service-timeout. |
--inactive-timeout <duration> | Get the HTTP RequestReadTimeout directive from both the WebUI and the HTTP service. |
--revert-inactive-timeout | Set value to system default for --inactive-timeout. |
--session-max-age <duration> | Get the HTTP SessionMaxAge directive from both WebUI and HTTP service. |
--revert-session-max-age | Set value to system default for --session-max-age. |
--httpd-controlpath-redirect <boolean> | Enable or disable WebUI redirection to the HTTP service. |
Note that while the OneFS S3 service uses HTTP, it is considered a tier-1 protocol, and as such is managed using its own isi s3 CLI command set and corresponding WebUI area. For example, the following CLI command forces the cluster to only accept encrypted HTTPS/SSL traffic on TCP port 9999 (rather than the default TCP port 9021):
# isi s3 settings global modify --https-only 1 –https-port 9921 # isi s3 settings global view HTTP Port: 9020 HTTPS Port: 9999 HTTPS only: Yes S3 Service Enabled: Yes
Additionally, you can entirely disable the S3 service with the following CLI command:
# isi services s3 disable The service 's3' has been disabled.
Or from the WebUI, under Protocols > S3 > Global settings:
Author: Nick Trimbee
OneFS Key Manager Rekey Support
Mon, 24 Jul 2023 19:16:34 -0000
|Read Time: 0 minutes
The OneFS key manager is a backend service that orchestrates the storage of sensitive information for PowerScale clusters. To satisfy Dell’s Secure Infrastructure Ready requirements and other public and private sector security mandates, the manager provides the ability to replace, or rekey, cryptographic keys.
The quintessential consumer of OneFS key management is data-at-rest encryption (DARE). Protecting sensitive data stored on the cluster with cryptography ensures that it’s guarded against theft, in the event that drives or nodes are removed from a PowerScale cluster. DARE is a requirement for federal and industry regulations, ensuring data is encrypted when it is stored. OneFS has provided DARE solutions for many years through secure encrypted drives (SEDs) and the OneFS key management system.
A 256-bit key (MK) encrypts the Key Manager Database (KMDB) for SED and cluster domains. In OneFS 9.2 and later, the MK for SEDs can either be stored off-cluster on a KMIP server or locally on a node (the legacy behavior).
However, there are a variety of other consumers of the OneFS key manager, in addition to DARE. These include services and protocols such as:
Service | Description |
---|---|
CELOG | Cluster event log |
CloudPools | Cluster tier to cloud service |
Electronic mail | |
FTP | File transfer protocol |
IPMI | Intelligent platform management interface for remote cluster console access |
JWT | JSON web tokens |
NDMP | Network data management protocol for cluster backups and DR |
Pstore | Active directory and Kerberos password store |
S3 | S3 object protocol |
SyncIQ | Cluster replication service |
SmartSync | OneFS push and pull replication cluster and cloud replication service |
SNMP | Simple network monitoring protocol |
SRS | Old Dell support remote cluster connectivity |
SSO | Single sign-on |
SupportAssist | Remote cluster connectivity to Dell Support |
OneFS 9.5 introduces a number of enhancements to the venerable key manager, including:
- The ability to rekey keystores. Rekey operation will generate a new MK and re-encrypt all entries stored with the new key.
- New CLI commands and WebUI options to perform a rekey operation or schedule key rotation on a time interval.
- New commands to monitor the progress and status of a rekey operation.
As such, OneFS 9.5 now provides the ability to rekey the MK, irrespective of where it is stored.
Note that when you are upgrading from an earlier OneFS release, the new rekey functionality is only available once the OneFS 9.5 upgrade has been committed.
Under the hood, each provider store in the key manager consists of secure backend storage and an MK. Entries are kept in a SQLite database or key-value store. A provider datastore uses its MK to encrypt all its entries within the store.
During the rekey process, the old MK is only deleted after a successful re-encryption with the new MK. If for any reason the process fails, the old MK is available and remains as the current MK. The rekey daemon retries the rekey every 15 minutes if the process fails.
The OneFS rekey process is as follows:
- A new MK is generated, and internal configuration is updated.
- Any entries in the provider store are decrypted and encrypted with the new MK.
- If the prior steps are successful, the previous MK is deleted.
To support the rekey process, the MK in OneFS 9.5 now has an ID associated with it. All entries have a new field referencing the MK ID.
During the rekey operation, there are two MK values with different IDs, and all entries in the database will associate which key they are encrypted by.
In OneFS 9.5, the rekey configuration and management is split between the cluster keys and the SED keys:
Rekey component | Detail |
---|---|
SED |
|
Cluster |
|
SED keys rekey
The SED key manager rekey operation can be managed through a DARE cluster’s CLI or WebUI, and it can either be automatically scheduled or run manually on demand. The following CLI syntax can be used to manually initiate a rekey:
# isi keymanager sed rekey start
Alternatively, to schedule a rekey operation, for example, to schedule a key rotation every two months:
# isi keymanager sed rekey modify --key-rotation=2m
The key manager status for SEDs can be viewed as follows:
# isi keymanager sed status Node Status Location Remote Key ID Key Creation Date Error Info(if any) ----------------------------------------------------------------------------- 1 LOCAL Local 1970-01-01T00:00:00 ----------------------------------------------------------------------------- Total: 1
Alternatively, from the WebUI, go to Access > Key Management > SED/Cluster Rekey, select Automatic rekey for SED keys, and configure the rekey frequency:
Note that for SED rekey operations, if a migration from local cluster key management to a KMIP server is in progress, the rekey process will begin once the migration is complete.
Cluster keys rekey
As mentioned previously, OneFS 9.5 also supports the rekey of cluster keystore domains. This cluster rekey operation is available through the CLI and the WebUI and may either be scheduled or run on demand. The available cluster domains can be queried by running the following CLI syntax:
# isi keymanager cluster status Domain Status Key Creation Date Error Info(if any) ---------------------------------------------------------- CELOG ACTIVE 2023-04-06T09:19:16 CERTSTORE ACTIVE 2023-04-06T09:19:16 CLOUDPOOLS ACTIVE 2023-04-06T09:19:16 EMAIL ACTIVE 2023-04-06T09:19:16 FTP ACTIVE 2023-04-06T09:19:16 IPMI_MGMT IN_PROGRESS 2023-04-06T09:19:16 JWT ACTIVE 2023-04-06T09:19:16 LHOTSE ACTIVE 2023-04-06T09:19:11 NDMP ACTIVE 2023-04-06T09:19:16 NETWORK ACTIVE 2023-04-06T09:19:16 PSTORE ACTIVE 2023-04-06T09:19:16 RICE ACTIVE 2023-04-06T09:19:16 S3 ACTIVE 2023-04-06T09:19:16 SIQ ACTIVE 2023-04-06T09:19:16 SNMP ACTIVE 2023-04-06T09:19:16 SRS ACTIVE 2023-04-06T09:19:16 SSO ACTIVE 2023-04-06T09:19:16 ---------------------------------------------------------- Total: 17
The rekey process generates a new key and re-encrypts the entries for the domain. The old key is then deleted.
Performance-wise, the rekey process does consume cluster resources (CPU and disk) as a result of the re-encryption phase, which is fairly write-intensive. As such, a good practice is to perform rekey operations outside of core business hours or during scheduled cluster maintenance windows.
During the rekey process, the old MK is only deleted once a successful re-encryption with the new MK has been confirmed. In the event of a rekey process failure, the old MK is available and remains as the current MK.
A rekey may be requested immediately or may be scheduled with a cadence. The rekey operation is available through the CLI and the WebUI. In the WebUI, go to Access > Key Management > SED/Cluster Rekey.
To start a rekey of the cluster domains immediately, from the CLI run the following syntax:
# isi keymanager cluster rekey start Are you sure you want to rekey the master passphrase? (yes/[no]):yes
Alternatively, from the WebUI, go to Access under the SED/Cluster Rekey tab, and click Rekey Now next to Cluster keys:
A scheduled rekey of the cluster keys (excluding the SED keys) can be configured from the CLI with the following syntax:
# isi keymanager cluster rekey modify –-key-rotation [YMWDhms]
Specify the frequency of the Key Rotation field as an integer, using Y for years, M for months, W for weeks, D for days, h for hours, m for minutes, and s for seconds. For example, the following command will schedule the cluster rekey operation to run every six weeks:
# isi keymanager cluster rekey view Rekey Time: 1970-01-01T00:00:00 Key Rotation: Never # isi keymanager cluster rekey modify --key-rotation 6W # isi keymanager cluster rekey view Rekey Time: 2023-04-28T18:38:45 Key Rotation: 6W
The rekey configuration can be easily reverted back to on demand from a schedule as follows:
# isi keymanager cluster rekey modify --key-rotation Never # isi keymanager cluster rekey view Rekey Time: 2023-04-28T18:38:45 Key Rotation: Never
Alternatively, from the WebUI, under the SED/Cluster Rekey tab, select the Automatic rekey for Cluster keys checkbox and specify the rekey frequency. For example:
In an event of a rekeying failure, a CELOG KeyManagerRekeyFailed or KeyManagerSedsRekeyFailed event is created. Since SED rekey is a node-local operation, the KeyManagerSedsRekeyFailed event information will also include which node experienced the failure.
Additionally, current cluster rekey status can also be queried with the following CLI command:
# isi keymanager cluster status Domain Status Key Creation Date Error Info(if any) ---------------------------------------------------------- CELOG ACTIVE 2023-04-06T09:19:16 CERTSTORE ACTIVE 2023-04-06T09:19:16 CLOUDPOOLS ACTIVE 2023-04-06T09:19:16 EMAIL ACTIVE 2023-04-06T09:19:16 FTP ACTIVE 2023-04-06T09:19:16 IPMI_MGMT ACTIVE 2023-04-06T09:19:16 JWT ACTIVE 2023-04-06T09:19:16 LHOTSE ACTIVE 2023-04-06T09:19:11 NDMP ACTIVE 2023-04-06T09:19:16 NETWORK ACTIVE 2023-04-06T09:19:16 PSTORE ACTIVE 2023-04-06T09:19:16 RICE ACTIVE 2023-04-06T09:19:16 S3 ACTIVE 2023-04-06T09:19:16 SIQ ACTIVE 2023-04-06T09:19:16 SNMP ACTIVE 2023-04-06T09:19:16 SRS ACTIVE 2023-04-06T09:19:16 SSO ACTIVE 2023-04-06T09:19:16 ---------------------------------------------------------- Total: 17
Or, for SEDs rekey status:
# isi keymanager sed status Node Status Location Remote Key ID Key Creation Date Error Info(if any) ----------------------------------------------------------------------------- 1 LOCAL Local 1970-01-01T00:00:00 2 LOCAL Local 1970-01-01T00:00:00 3 LOCAL Local 1970-01-01T00:00:00 4 LOCAL Local 1970-01-01T00:00:00 ----------------------------------------------------------------------------- Total: 4
The rekey process also outputs to the /var/log/isi_km_d.log file, which is a useful source for additional troubleshooting.
If an error in rekey occurs, the previous MK is not deleted, so entries in the provider store can still be created and read as normal. The key manager daemon will retry the rekey operation in the background every 15 minutes until it succeeds.
Author: Nick Trimbee