OneFS System Configuration Auditing
Thu, 18 Apr 2024 04:55:18 -0000
|Read Time: 0 minutes
OneFS auditing can detect potential sources of data loss, fraud, inappropriate entitlements, access attempts that should not occur, and a range of other anomalies that are indicators of risk. This can be especially useful when the audit associates data access with specific user identities.
In the interests of data security, OneFS provides “chain of custody” auditing by logging specific activity on the cluster. This includes OneFS configuration changes plus NFS, SMB, and HDFS client protocol activity which are required for organizational IT security compliance, as mandated by regulatory bodies like HIPAA, SOX, FISMA, MPAA, and more.
OneFS auditing uses Dell’s Common Event Enabler (CEE) to provide compatibility with external audit applications. A cluster can write audit events across up to five CEE servers per node in a parallel, load-balanced configuration. This allows OneFS to deliver an end to end, enterprise grade audit solution which efficiently integrates with third party solutions like Varonis DatAdvantage.
The following diagram outlines the basic architecture of OneFS audit:
Both system configuration changes, as well as protocol activity, can be easily audited on a PowerScale cluster. However, the protocol path is greyed out above, since it is outside the focus of this article. More information on OneFS protocol auditing can be found here.
As illustrated above, the OneFS audit framework is centered around three main services.
Service | Description |
isi_audit_cee | Service allowing OneFS to support third-party auditing applications. The main method of accessing protocol audit data from OneFS is through a third-party auditing application. |
isi_audit_d | Responsible for per-node audit queues and managing the data store for those queues. It provides a protocol on which clients may produce event payloads within a given context. It establishes a Unix domain socket for queue producers and handles writing and rotation of log files in /ifs/.ifsvar/audit/logs/node###/{config,protocol}/*. |
isi_audit_syslog | Daemon providing forwarding of audit config and protocol events to syslog. |
The basic configuration auditing workflow sees a cluster config change request come in via either the OneFS CLI, WebUI or platform API. The API handler infrastructure passes this request to the isi_audit_d service which intercepts it as a client thread and adds it to the audit queue. It is then processed and passed via a backend thread and written to the audit log files (IFS) as appropriate.
If audit syslog forwarding has been configured, IFS also passes the event to the isi_audit_syslog daemon, where a supervisor process instructs a writer thread to send it to the syslog which in turn updates its pertinent /var/log/ logfiles.
Similarly, if Common Event Enabler (CEE) forwarding has been enabled, IFS will also pass the request to the isi_audit_cee service where a delivery worker threads will intercept it and send the event to the CEE server pool. The isi_audit_cee heartbeat task makes CEE servers available for audit event delivery. Only after a CEE server has received a successful heartbeat will audit events be delivered to it. Every ten seconds, the heartbeat task wakes up and sends each CEE server in the configuration a heartbeat. While CEE servers are available and events are in memory, an attempt will be made to deliver these. Shutdown will only save audit log position if all the events are delivered to CEE since audit should not lose events. It isn't critical that all events are delivered at shutdown since any unsaved events can be resent to CEE on the next start of isi_audit_cee since CEE handles duplicates.
Within OneFS, all audit data is organized by topic and is securely stored in the file system.
# isi audit topics list
Name Max Cached Messages
-----------------------------
protocol 2048
config 1024
-----------------------------
Total: 2
Auditing can detect a variety of potential sources of data loss. These include unauthorized access attempts, inappropriate entitlements, plus a bevy of other fraudulent activities that plague organizations across the gamut of industries. Enterprises are increasingly required to comply with stringent regulatory mandates developed to protect against these sources of data theft and loss.
OneFS system configuration auditing is designed to track and record all configuration events that are handled by the API through the command-line interface (CLI).
# isi audit topics view config
Name: config
Max Cached Messages: 1024
Once enabled, system configuration auditing requires no additional configuration, and auditing events are automatically stored in the config audit topic directories. Audit access and management is governed by the ‘ISI_PRIV_AUDIT’ RBAC privilege, and OneFS provides a default ‘AuditAdmin’ role for this purpose.
Audit events are stored in a binary file under /ifs/.ifsvar/audit/logs. The logs automatically roll over to a new file after the size reaches 1 GB. The audit logs are consumable by auditing applications that support the Dell Common Event Enabler (CEE).
OneFS audit topics and settings can easily be viewed and modified. For example, to increase the configuration auditing maximum cached messages threshold to 2048 from the CLI:
# isi audit topics modify config --max-cached-messages 2048
# isi audit topics view config
Name: config
Max Cached Messages: 2048
Audit configuration can also be modified or viewed per access zone and/or topic.
Operation | CLI Syntax | Method and URI |
Get audit settings | isi audit settings view | GET <cluster-ip:port>/platform/3/audit/settings |
Modify audit settings | isi audit settings modify … | PUT <cluster-ip:port>/platform/3/audit/settings |
View JSON schema for this resource, including query parameters and object properties info. |
| GET <cluster-ip:port>/platform/3/audit/settings?describe |
View JSON schema for this resource, including query parameters and object properties info. |
| GET <cluster-ip:port>/platform/1/audit/topics?describe |
Configuration auditing can be enabled on a cluster from either the CLI or platform API. The current global audit configuration can be viewed as follows:
1# isi audit settings global view
Protocol Auditing Enabled: No
Audited Zones: -
CEE Server URIs: -
Hostname:
Config Auditing Enabled: No
Config Syslog Enabled: No
Config Syslog Servers: -
Config Syslog TLS Enabled: No
Config Syslog Certificate ID:
Protocol Syslog Servers: -
Protocol Syslog TLS Enabled: No
Protocol Syslog Certificate ID:
System Syslog Enabled: No
System Syslog Servers: -
System Syslog TLS Enabled: No
System Syslog Certificate ID:
Auto Purging Enabled: No
Retention Period: 180
System Auditing Enabled: No
In this case, configuration auditing is disabled – its default setting. The following CLI syntax will enable (and verify) configuration auditing across the cluster:
# isi audit settings global modify --config-auditing-enabled 1
# isi audit settings global view | grep -i 'config audit'
Config Auditing Enabled: Yes
In the next article, we’ll look at the config audit management, event viewing, and troubleshooting.
To enable configuration change audit redirection to syslog:
# isi audit settings global modify --config-auditing-enabled true
# isi audit settings global modify --config-syslog-enabled true
# isi audit settings global view | grep -i 'config audit'
Config Auditing Enabled: Yes
Similarly, to disable configuration change audit redirection to syslog:
# isi audit settings global modify --config-syslog-enabled false
# isi audit settings global modify --config-auditing-enabled false
configure audit
2.
#isi audit setting modify --add-cee-server-uris='http://seavee5.west.isilon.com:12228/cee'
4.
# isi audit settings modify --add-audited-zones=auditgti
4' if you don't want audit that much
# isi audit setting modify --remove-audited-zones=System
config zone
3.
#isi zone zones create --all-auth-providers=true --audit-failure=all --audit-success=all --path=/ifs/data --name=auditgti
3'. if you dont' want to audit that much
#isi zone zones create --all-auth-providers=true --audit-failure=read,logon --audit-success=write,delete --path=/ifs/data --name=auditgti
network pool
5.
#isi network create pool --name=subnet0:auditpool --access-zone=auditgit --iface=<your interface> --range=<your range>
5' you can also audit System by default, so this step can be ignored
other settings
#isi audit setting modify --hostname="<any name you want really, this just gets inserted into the payload>"
#isi audit setting modify --cee-log-time="Protocol@1900-01-01 00:00:01"
The platform API can also be used to configure and manage auditing. For example, to enable configuration auditing on a cluster:
PUT /platform/1/audit/settings
Authorization: Basic QWxhZGRpbjpvcGVuIHN1c2FtZQ==
{
'config_auditing_enabled': True
}
Response example
The HTTP ‘204 response code from the cluster indicates that the request was successful, and that configuration auditing is now enabled on the cluster. No message body is returned for this request.
204 No Content
Content-type: text/plain,
Allow: 'GET, PUT, HEAD'
Similarly, to modify the config audit topic’s maximum cached messages threshold to a value of ‘1000’ via the API:
PUT /1/audit/topics/config
Authorization: Basic QWxhZGRpbjpvcGVuIHN1c2FtZQ==
{
"max_cached_messages": 1000
}
Again, no message body is returned from OneFS for this request.
204 No Content
Content-type: text/plain,
Allow: 'GET, PUT, HEAD'
Note that, in the unlikely event that a cluster experiences an outage during which it loses quorum, auditing will be suspended until it is regained. Events similar to the following will be written to the /var/log/audit_d.log file:
940b5c700]: Lost quorum! Audit logging will be disabled until /ifs is writeable again.
2023-08-28T15:37:32.132780+00:00 <1.6> TME-1(id1) isi_audit_d[6495]: [0x345940b5c700]: Regained quorum. Logging resuming.
When it comes to reading audit events on the cluster, OneFS natively provides the handy ‘isi_audit_viewer’ utility. For example, the following audit viewer output shows the events logged when the cluster admin added the ‘/ifs/tmp’ path to the SmartDedupe configuration, and created a new user named ‘test’1’:
# isi_audit_viewer
[0: Tue Aug 29 23:01:16 2023] {"id":"f54a6bec-46bf-11ee-920d-0060486e0a26","timestamp":1693350076315499,"payload":{"user":{"token": {"UID":0, "GID":0, "SID": "SID:S-1-22-1-0", "GSID": "SID:S-1-22-2-0", "GROUPS": ["SID:S-1-5-11", "GID:5", "GID:10", "GID:20", "GID:70"], "protocol": 17, "zone id": 1, "client": "10.135.6.255", "local": "10.219.64.11" }},"uri":"/1/dedupe/settings","method":"PUT","args":{}
,"body":{"paths":["/ifs/tmp"]}
}}
[1: Tue Aug 29 23:01:16 2023] {"id":"f54a6bec-46bf-11ee-920d-0060486e0a26","timestamp":1693350076391422,"payload":{"status":204,"statusmsg":"No Content","body":{}}}
[2: Tue Aug 29 23:03:43 2023] {"id":"4cfce7a5-46c0-11ee-920d-0060486e0a26","timestamp":1693350223446993,"payload":{"user":{"token": {"UID":0, "GID":0, "SID": "SID:S-1-22-1-0", "GSID": "SID:S-1-22-2-0", "GROUPS": ["SID:S-1-5-11", "GID:5", "GID:10", "GID:20", "GID:70"], "protocol": 17, "zone id": 1, "client": "10.135.6.255", "local": "10.219.64.11" }},"uri":"/18/auth/users","method":"POST","args":{}
,"body":{"name":"test1"}
}}
[3: Tue Aug 29 23:03:43 2023] {"id":"4cfce7a5-46c0-11ee-920d-0060486e0a26","timestamp":1693350223507797,"payload":{"status":201,"statusmsg":"Created","body":{"id":"SID:S-1-5-21-593535466-4266055735-3901207217-1000"}
}}
The audit log entries, such as those above, typically comprise the following components:
- Timestamp (Human readable)
- Unique Entry ID
- Timestamp (Unix Epoch Time)
- Node Number
- The user tokens of the person executing the command
- User persona (Unix/Windows)
- Primary group persona (Unix/Windows)
- Supplemental group personas (Unix/Windows)
- RBAC privileges of the person executing the command
- Interface used to generate the command
- 10 = PAPI / WebUI
- 16 = Console
- 17 = SSH
- Access Zone that the command was executed against
- Where the user connected from
- The local node address where the command was executed
- Command
- Command arguments
- Command body
The ‘isi_audit_viewer’ utility automatically reads the ‘config’ log topic by default, but can also be used read the ‘protocol’ log topic too. Its CLI command syntax is as follows:
# isi_audit_viewer -h
Usage: isi_audit_viewer [ -n <nodeid> | -t <topic> | -s <starttime>|
-e <endtime> | -v ]
-n <nodeid> : Specify node id to browse (default: local node)
-t <topic> : Choose topic to browse.
Topics are "config" and "protocol" (default: "config")
-s <start> : Browse audit logs starting at <starttime>
-e <end> : Browse audit logs ending at <endtime>
-v verbose : Prints out start / end time range before printing
records
Note that, on large clusters where there is heavy (up to the 100,000’s) of audit writes, when running the isi_audit_viewer utility across the cluster with ‘isi_for_array’, it can potentially lead to memory starvation and other issues – especially if outputting to a directory under /ifs. As such, consider directing the output to a non-IFS location such as /var/temp. Also, the isi_audit_viewer ‘-s’ (start time) and ‘-e’ (end time) flags can be used to limit a search (for 1-5 minutes), helping reduce the size of data.
In addition to reading audit events, the view is also a useful tool to assist with troubleshoot any auditing issues. Additionally, any errors that are encountered while processing audit events, and when delivering them to an external CEE server, are written to the log file ‘/var/log/isi_audit_cee.log’. Additionally, the protocol specific logs will contain any issues the audit filter has collecting while auditing events.