To enable granular HTTP security configuration, OneFS provides an option to disable nonessential HTTP components selectively. This can help reduce the overall attack surface of your infrastructure. Disabling a specific component’s service still allows other essential services on the cluster to continue to run unimpeded. In OneFS 9.4 and later, you can disable the following nonessential HTTP services:

You can enable or disable each of these services independently, using the CLI or platform API, if you have a user account with the ISI_PRIV_HTTP RBAC privilege.

You can use the isi http services CLI command set to view and modify the nonessential HTTP services:

# isi http services list
ID                     Enabled
------------------------------
Platform-API-External Yes
PowerScaleUI          Yes
RAN                   Yes
RemoteService         Yes
SWIFT                 No
------------------------------
Total: 5

For example, you can easily disable remote HTTP access to the OneFS /ifs namespace as follows:

# isi http services modify RAN --enabled=0

You are about to modify the service RAN. Are you sure? (yes/[no]): yes

Similarly, you can also use the WebUI to view and edit a subset of the HTTP configuration settings, by navigating to Protocols > HTTP settings:

That said, the implications and impact of disabling each of the services is as follows:

You can use the CLI command isi http settings view to display the OneFS HTTP configuration:

# isi http settings view
            Access Control: No
      Basic Authentication: No
    WebHDFS Ran HTTPS Port: 8443
                        Dav: No
         Enable Access Log: Yes
                      HTTPS: No
 Integrated Authentication: No
               Server Root: /ifs
                    Service: disabled
           Service Timeout: 8m20s
          Inactive Timeout: 15m
           Session Max Age: 4H
Httpd Controlpath Redirect: No

Similarly, you can manage and change the HTTP configuration using the isi http settings modify CLI command.

For example, to reduce the maximum session age from four to two hours:

# isi http settings view | grep -i age
           Session Max Age: 4H
# isi http settings modify --session-max-age=2H
# isi http settings view | grep -i age
           Session Max Age: 2H

The full set of configuration options for isi http settings includes:

Note that while the OneFS S3 service uses HTTP, it is considered a tier-1 protocol, and as such is managed using its own isi s3 CLI command set and corresponding WebUI area. For example, the following CLI command forces the cluster to only accept encrypted HTTPS/SSL traffic on TCP port 9999 (rather than the default TCP port 9021):

# isi s3 settings global modify --https-only 1 –https-port 9921
# isi s3 settings global view
         HTTP Port: 9020
        HTTPS Port: 9999
        HTTPS only: Yes
S3 Service Enabled: Yes

Additionally, you can entirely disable the S3 service with the following CLI command:

# isi services s3 disable
The service 's3' has been disabled.

Or from the WebUI, under Protocols > S3 > Global settings:

 Author: Nick Trimbee

Another security enhancement that OneFS 9.5 and later releases brings to the table is the ability to configure 1GbE NIC ports dedicated to cluster management on the PowerScale F900, F710, F600, F210, and F200 all-flash storage nodes and P100 and B100 accelerators. Since these platforms were released, customers have been requesting the ability to activate the 1GbE NIC ports so that the node management activity and front end protocol traffic can be separated on physically distinct interfaces.

For background, since their introduction, the F600 and F900 have shipped with a quad port 1GbE rNDC (rack Converged Network Daughter Card) adapter. However, these 1GbE ports were non-functional and unsupported in OneFS releases prior to 9.5. As such, the node management and front-end traffic was co-mingled on the front-end interface.

In OneFS 9.5 and later, 1GbE network ports are now supported on all of the PowerScale PowerEdge based platforms for the purposes of node management, and are physically separate from the other network interfaces. Specifically, this enhancement applies to the F900, F600, F200 all-flash nodes, and P100 and B100 accelerators.

Under the hood, OneFS has been updated to recognize the 1GbE rNDC NIC ports as usable for a management interface. Note that the focus of this enhancement is on factory enablement and support for existing F600 customers that have the unused 1GbE rNDC hardware. This functionality has also been back-ported to OneFS 9.4.0.3 and later RUPs. Since the introduction of this feature, there have been several requests raised about field upgrades, but that use case is separate and will be addressed in a later release through scripts, updates of node receipts, procedures, and so on.

Architecturally, aside from some device driver and accounting work, no substantial changes were required to the underlying OneFS or platform architecture to implement this feature. This means that in addition to activating the rNDC, OneFS now supports the relocated front-end NIC in PCI slots 2 or 3 for the F200, B100, and P100.

OneFS 9.5 and later recognizes the 1GbE rNDC as usable for the management interface in the OneFS Wizard, in the same way it always has for the H-series and A-series chassis-based nodes.

All four ports in the 1GbE NIC are active, and for the Broadcom board, the interfaces are initialized and reported as bge0, bge1, bge2, and bge3.

The pciconf CLI utility can be used to determine whether the rNDC NIC is present in a node. If it is, a variety of identification and configuration details are displayed. For example, let’s look at the following output from a Broadcom rNDC NIC in an F200 node:

# pciconf -lvV pci0:24:0:0

bge2@pci0:24:0:0: class=0x020000 card=0x1f5b1028 chip=0x165f14e4 rev=0x00 hdr=0x00
      class       = network
      subclass    = ethernet
      VPD ident   = ‘Broadcom NetXtreme Gigabit Ethernet’
      VPD ro PN   = ‘BCM95720’
      VPD ro MN   = ‘1028’
      VPD ro V0   = ‘FFV7.2.14’
      VPD ro V1   = ‘DSV1028VPDR.VER1.0’
      VPD ro V2   = ‘NPY2’
      VPD ro V3   = ‘PMT1’
      VPD ro V4   = ‘NMVBroadcom Corp’
      VPD ro V5   = ‘DTINIC’
      VPD ro V6   = ‘DCM1001008d452101000d45’

We can use the ifconfig CLI utility to determine the specific IP/interface mapping on the Broadcom rNDC interface. For example:

# ifconfig bge0
 TME-1: bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 TME-1:      ether 00:60:16:9e:X:X
 TME-1:      inet 10.11.12.13 netmask 0xffffff00 broadcast 10.11.12.255 zone 1
 TME-1:      inet 10.11.12.13 netmask 0xffffff00 broadcast 10.11.12.255 zone 0
 TME-1:      media: Ethernet autoselect (1000baseT <full-duplex>)
 TME-1:      status: active

In this output, the first IP address of the management interface’s pool is bound to bge0, which is the first port on the Broadcom rNDC NIC.

We can use the isi network pools CLI command to determine the corresponding interface. Within the system zone, the management interface is allocated an address from the configured IP range within its associated interface pool. For example:

# isi network pools list
ID                      SC Zone                  IP Ranges                   Allocation Method
----------------------------------------------------------------------------------------------
groupnet0.mgt.mgt       cluster_mgt_isln.com     10.11.12.13-10.11.12.20     static
# isi network pools view groupnet0.mgt.mgt | grep -i ifaces
               Ifaces: 1:mgmt-1, 2:mgmt-1, 3:mgmt-1, 4:mgmt-1, 5:mgmt-1

Or from the WebUI, under Network configuration > External network:

Drilling down into the mgt pool details shows the 1GbE management interfaces as the pool interface members:

Note that the 1GbE rNDC network ports are solely intended as cluster management interfaces. As such, they are not supported for use with regular front-end data traffic.

The F900 and F600 nodes already ship with a four port 1GbE rNDC NIC installed. However, the F200, B100, and P100 platform configurations have also been updated to include a quad port 1GbE rNDC card. These new configurations have been shipping by default since January 2023. This required relocating the front end network’s 25GbE NIC (Mellanox CX4) to PCI slot 2 in the motherboard. Additionally, the OneFS updates needed for this feature have also now allowed the F200 platform to be offered with a 100GbE option too. The 100GbE option uses a Mellanox CX6 NIC in place of the CX4 in slot 2.

With this 1GbE management interface enhancement, the same quad-port rNDC card (typically the Broadcom 5720) that has been shipped in the F900 and F600 since their introduction, is now included in the F200, B100 and P100 nodes as well. All four 1GbE rNDC ports are enabled and active under OneFS 9.5 and later, too.

Node port ordering continues to follow the standard, increasing numerically from left to right. However, be aware that the port labels are not visible externally because they are obscured by the enclosure’s sheet metal.

The following back-of-chassis hardware images show the new placements of the NICs in the various F-series and accelerator platforms:

F600

F900

For both the F600 and F900, the NIC placement remains unchanged, because these nodes have always shipped with the 1GbE quad port in the rNDC slot since their launch.

F200

The F200 sees its front-end NIC moved to slot 3, freeing up the rNDC slot for the quad-port 1GbE Broadcom 5720.

Because the B100 backup accelerator has a fibre-channel card in slot 2, it sees its front-end NIC moved to slot 3, freeing up the rNDC slot for the quad-port 1GbE Broadcom 5720.

Finally, the P100 accelerator sees its front-end NIC moved to slot 3, freeing up the rNDC slot for the quad-port 1GbE Broadcom 5720.

Note that, while there is currently no field hardware upgrade process for adding rNDC cards to legacy F200 nodes or B100 and P100 accelerators, this will be addressed in a future release.

Author: Nick Trimbee

As we’ve seen over the course of the last several articles, OneFS 9.5 delivers a wealth of security focused features. These span the realms of core file system, protocols, data services, platform, and peripherals. Among these security enhancements is the ability to manually or automatically disable a cluster’s USB ports from either the CLI, platform API, or by activating a security hardening policy.

In support of this functionality, the basic USB port control architecture is as follows:

To facilitate this, OneFS 9.5 and subsequent releases see the addition of a new gconfig variable, ‘usb_ports_disabled’, in ‘security_config’, specifically to track the status of USB Ports on a cluster. On receiving an admin request either from the CLI or the platform API handler to disable the USB port, OneFS modifies the security config parameter in gconfig. For example:

# isi_gconfig -t security_config | grep -i usb

usb_ports_disabled (bool) = true

Under the hood, the MCP (master control process) daemon watches for any changes to the ‘isi_security.gcfg’ security config file on the cluster. If the value for the ‘usb_ports_disabled’ variable in the ‘isi_security.gcfg’ file is updated, then MCP executes the ‘isi_config_usb’ utility to enact the desired change. Note that because ‘isi_config_usb’ operates per-node but the MCP actions are global (executed cluster wide), isi_config_usb is invoked across each node by a Python script to enable or disable the cluster’s USB Ports.

The USB Ports enable/disable feature is only supported on PowerScale F900, F600, F200, H700/7000, and A300/3000 clusters running OneFS 9.5 and later, and PowerScale F710 and F210 running OneFS 9.7 or later.

In OneFS 9.5 and later, USB port control can be manually configured from either the CLI or platform API.

Note that there is no WebUI option at this time.

The following table lists the CLI and platform API configuration options for USB port control in OneFS 9.5 and later:

For example:

# isi security settings view | grep -i usb
      USB Ports Disabled: No
# isi security settings modify --usb-ports-disabled=True
# isi security settings view | grep -i usb
      USB Ports Disabled: Yes

Similarly, to re-enable a cluster’s USB ports:

# isi security settings modify --usb-ports-disabled=False
# isi security settings view | grep -i usb
      USB Ports Disabled: No

Note that a user account with the OneFS ISI_PRIV_CLUSTER RBAC privilege is required to configure USB port changes on a cluster.

In addition to the ‘isi security settings’ CLI command, there is also a node-local CLI utility:

# whereis isi_config_usb
isi_config_usb: /usr/bin/isi_hwtools/isi_config_usb

As mentioned previously, ‘isi security settings’ acts globally on a cluster, using ‘isi_config_usb’ to effect its changes on each node.

Alternatively, cluster USB ports can also be enabled and disabled using the OneFS platform API with the following endpoints:

For example:

# curl -k -u <username>:<passwd> https://localhost:8080/platform/security/settings”
 
 {
 "settings" :
 {
 "fips_mode_enabled" : false,
 "restricted_shell_enabled" : false,
 "usb_ports_disabled" : true
 }
 }

In addition to manual configuration, the USB ports are automatically disabled if the STIG security hardening profile is applied to a cluster. 

This is governed by the following section of XML code in the isi_hardening configuration file, which can be found at /etc/isi_hardening/profiles/isi_hardening.xml:

<CONFIG_ITEM id ="isi_usb_ports" version = "1">
              <PapiOperation>
                           <DO>
                                        <URI>/security/settings</URI>
                                        <BODY>{"usb_ports_disabled": true}</BODY>
                                        <KEY>settings</KEY>
                           </DO>
                           <UNDO>
                                        <URI>/security/settings</URI>
                                        <BODY>{"usb_ports_disabled": false}</BODY>
                                        <KEY>settings</KEY>
                           </UNDO>
                           <ACTION_SCOPE>CLUSTER</ACTION_SCOPE>
                           <IGNORE>FALSE</IGNORE>
              </PapiOperation>
 </CONFIG_ITEM>

The ‘isi_config_usb’ CLI utility can be used to display the USB port status on a subset of nodes. For example:

# isi_config_usb --nodes 1-10 --mode display
    Node   |   Current  |  Pending
-----------------------------------
    TME-9  |   UNSUP    | INFO: This platform is not supported to run this script.
    TME-8  |   UNSUP    | INFO: This platform is not supported to run this script.
    TME-1  |     On     |
    TME-3  |     On     |
    TME-2  |     On     |
   TME-10  |     On     |
    TME-7  |   AllOn    |
    TME-5  |   AllOn    |
    TME-6  |   AllOn    |
Unable to connect: TME-4

Note: In addition to port status, the output identifies any nodes that do not support USB port control (nodes 8 and 9 above) or that are unreachable (node 4 above).

When investigating or troubleshooting issues with USB port control, the following log files are the first places to check:

Author: Nick Trimbee

OneFS auditing can detect potential sources of data loss, fraud, inappropriate entitlements, access attempts that should not occur, and a range of other anomalies that are indicators of risk. This can be especially useful when the audit associates data access with specific user identities. 

In the interests of data security, OneFS provides “chain of custody” auditing by logging specific activity on the cluster. This includes OneFS configuration changes plus NFS, SMB, and HDFS client protocol activity which are required for organizational IT security compliance, as mandated by regulatory bodies like HIPAA, SOX, FISMA, MPAA, and more. 

OneFS auditing uses Dell’s Common Event Enabler (CEE) to provide compatibility with external audit applications. A cluster can write audit events across up to five CEE servers per node in a parallel, load-balanced configuration. This allows OneFS to deliver an end to end, enterprise grade audit solution which efficiently integrates with third party solutions like Varonis DatAdvantage. 

The following diagram outlines the basic architecture of OneFS audit: 

  Both system configuration changes, as well as protocol activity, can be easily audited on a PowerScale cluster. However, the protocol path is greyed out above, since it is outside the focus of this article. More information on OneFS protocol auditing can be found here. 

As illustrated above, the OneFS audit framework is centered around three main services. 

 The basic configuration auditing workflow sees a cluster config change request come in via either the OneFS CLI, WebUI or platform API. The API handler infrastructure passes this request to the isi_audit_d service which intercepts it as a client thread and adds it to the audit queue. It is then processed and passed via a backend thread and written to the audit log files (IFS) as appropriate.

If audit syslog forwarding has been configured, IFS also passes the event to the isi_audit_syslog daemon, where a supervisor process instructs a writer thread to send it to the syslog which in turn updates its pertinent /var/log/ logfiles.  

Similarly, if Common Event Enabler (CEE) forwarding has been enabled, IFS will also pass the request to the isi_audit_cee service where a delivery worker threads will intercept it and send the event to the CEE server pool. The isi_audit_cee heartbeat task makes CEE servers available for audit event delivery. Only after a CEE server has received a successful heartbeat will audit events be delivered to it. Every ten seconds, the heartbeat task wakes up and sends each CEE server in the configuration a heartbeat. While CEE servers are available and events are in memory, an attempt will be made to deliver these. Shutdown will only save audit log position if all the events are delivered to CEE since audit should not lose events. It isn't critical that all events are delivered at shutdown since any unsaved events can be resent to CEE on the next start of isi_audit_cee since CEE handles duplicates.

 Within OneFS, all audit data is organized by topic and is securely stored in the file system.

# isi audit topics list

Name     Max Cached Messages

-----------------------------

protocol 2048

config   1024

-----------------------------

Total: 2

Auditing can detect a variety of potential sources of data loss. These include unauthorized access attempts, inappropriate entitlements, plus a bevy of other fraudulent activities that plague organizations across the gamut of industries. Enterprises are increasingly required to comply with stringent regulatory mandates developed to protect against these sources of data theft and loss.

OneFS system configuration auditing is designed to track and record all configuration events that are handled by the API through the command-line interface (CLI).  

# isi audit topics view config

               Name: config

Max Cached Messages: 1024

Once enabled, system configuration auditing requires no additional configuration, and auditing events are automatically stored in the config audit topic directories. Audit access and management is governed by the ‘ISI_PRIV_AUDIT’ RBAC privilege, and OneFS provides a default ‘AuditAdmin’ role for this purpose.

Audit events are stored in a binary file under /ifs/.ifsvar/audit/logs. The logs automatically roll over to a new file after the size reaches 1 GB. The audit logs are consumable by auditing applications that support the Dell Common Event Enabler (CEE).

OneFS audit topics and settings can easily be viewed and modified. For example, to increase the configuration auditing maximum cached messages threshold to 2048 from the CLI:

# isi audit topics modify config --max-cached-messages 2048

# isi audit topics view config

               Name: config

Max Cached Messages: 2048

Audit configuration can also be modified or viewed per access zone and/or topic.

Configuration auditing can be enabled on a cluster from either the CLI or platform API. The current global audit configuration can be viewed as follows:

1# isi audit settings global view

     Protocol Auditing Enabled: No

                 Audited Zones: -

               CEE Server URIs: -

                       Hostname:

       Config Auditing Enabled: No

         Config Syslog Enabled: No

         Config Syslog Servers: -

     Config Syslog TLS Enabled: No

  Config Syslog Certificate ID:

       Protocol Syslog Servers: -

   Protocol Syslog TLS Enabled: No

Protocol Syslog Certificate ID:

         System Syslog Enabled: No

         System Syslog Servers: -

     System Syslog TLS Enabled: No

  System Syslog Certificate ID:

          Auto Purging Enabled: No

              Retention Period: 180

       System Auditing Enabled: No

In this case, configuration auditing is disabled – its default setting. The following CLI syntax will enable (and verify) configuration auditing across the cluster:

# isi audit settings global modify --config-auditing-enabled 1

# isi audit settings global view | grep -i 'config audit'

       Config Auditing Enabled: Yes

In the next article, we’ll look at the config audit management, event viewing, and troubleshooting.

To enable configuration change audit redirection to syslog:

# isi audit settings global modify --config-auditing-enabled true

# isi audit settings global modify --config-syslog-enabled true

# isi audit settings global view | grep -i 'config audit'

       Config Auditing Enabled: Yes

Similarly, to disable configuration change audit redirection to syslog:

# isi audit settings global modify --config-syslog-enabled false

# isi audit settings global modify --config-auditing-enabled false

configure audit

2.

#isi audit setting modify --add-cee-server-uris='http://seavee5.west.isilon.com:12228/cee'

4.

# isi audit settings modify --add-audited-zones=auditgti

4' if you don't want audit that much

# isi audit setting modify --remove-audited-zones=System

config zone

3.

#isi zone zones create --all-auth-providers=true --audit-failure=all --audit-success=all --path=/ifs/data --name=auditgti

3'. if you dont' want to audit that much

#isi zone zones create --all-auth-providers=true --audit-failure=read,logon --audit-success=write,delete --path=/ifs/data --name=auditgti

network pool

5.

#isi network create pool --name=subnet0:auditpool --access-zone=auditgit --iface=<your interface> --range=<your range>  

5' you can also audit System by default, so this step can be ignored  

other settings

#isi audit setting modify --hostname="<any name you want really, this just gets inserted into the payload>"

#isi audit setting modify --cee-log-time="Protocol@1900-01-01 00:00:01"

The platform API can also be used to configure and manage auditing. For example, to enable configuration auditing on a cluster:

PUT /platform/1/audit/settings

Authorization: Basic QWxhZGRpbjpvcGVuIHN1c2FtZQ==

{

'config_auditing_enabled': True

}

Response example

The HTTP ‘204 response code from the cluster indicates that the request was successful, and that configuration auditing is now enabled on the cluster. No message body is returned for this request.

204 No Content

Content-type: text/plain,  

Allow: 'GET, PUT, HEAD'

Similarly, to modify the config audit topic’s maximum cached messages threshold to a value of ‘1000’ via the API:

PUT /1/audit/topics/config

Authorization: Basic QWxhZGRpbjpvcGVuIHN1c2FtZQ==

    {

         "max_cached_messages": 1000

    }

Again, no message body is returned from OneFS for this request.

204 No Content  

Content-type: text/plain,  

Allow: 'GET, PUT, HEAD'

Note that, in the unlikely event that a cluster experiences an outage during which it loses quorum, auditing will be suspended until it is regained. Events similar to the following will be written to the /var/log/audit_d.log file:

940b5c700]: Lost quorum! Audit logging will be disabled until /ifs is writeable again.

2023-08-28T15:37:32.132780+00:00 <1.6> TME-1(id1) isi_audit_d[6495]: [0x345940b5c700]: Regained quorum. Logging resuming.

When it comes to reading audit events on the cluster, OneFS natively provides the handy ‘isi_audit_viewer’ utility. For example, the following audit viewer output shows the events logged when the cluster admin added the ‘/ifs/tmp’ path to the SmartDedupe configuration, and created a new user named ‘test’1’:

# isi_audit_viewer

[0: Tue Aug 29 23:01:16 2023] {"id":"f54a6bec-46bf-11ee-920d-0060486e0a26","timestamp":1693350076315499,"payload":{"user":{"token": {"UID":0, "GID":0, "SID": "SID:S-1-22-1-0", "GSID": "SID:S-1-22-2-0", "GROUPS": ["SID:S-1-5-11", "GID:5", "GID:10", "GID:20", "GID:70"], "protocol": 17, "zone id": 1, "client": "10.135.6.255", "local": "10.219.64.11" }},"uri":"/1/dedupe/settings","method":"PUT","args":{}

,"body":{"paths":["/ifs/tmp"]}

}}

[1: Tue Aug 29 23:01:16 2023] {"id":"f54a6bec-46bf-11ee-920d-0060486e0a26","timestamp":1693350076391422,"payload":{"status":204,"statusmsg":"No Content","body":{}}}

[2: Tue Aug 29 23:03:43 2023] {"id":"4cfce7a5-46c0-11ee-920d-0060486e0a26","timestamp":1693350223446993,"payload":{"user":{"token": {"UID":0, "GID":0, "SID": "SID:S-1-22-1-0", "GSID": "SID:S-1-22-2-0", "GROUPS": ["SID:S-1-5-11", "GID:5", "GID:10", "GID:20", "GID:70"], "protocol": 17, "zone id": 1, "client": "10.135.6.255", "local": "10.219.64.11" }},"uri":"/18/auth/users","method":"POST","args":{}

,"body":{"name":"test1"}

}}

[3: Tue Aug 29 23:03:43 2023] {"id":"4cfce7a5-46c0-11ee-920d-0060486e0a26","timestamp":1693350223507797,"payload":{"status":201,"statusmsg":"Created","body":{"id":"SID:S-1-5-21-593535466-4266055735-3901207217-1000"}

}}

The audit log entries, such as those above, typically comprise the following components:

1. Timestamp (Human readable)
2. Unique Entry ID
3. Timestamp (Unix Epoch Time)
4. Node Number
5. The user tokens of the person executing the command
   1. User persona (Unix/Windows)
   2. Primary group persona (Unix/Windows)
   3. Supplemental group personas (Unix/Windows)
   4. RBAC privileges of the person executing the command
6. Interface used to generate the command
   1. 10 = PAPI / WebUI
   2. 16 = Console
   3. 17 = SSH
7. Access Zone that the command was executed against
8. Where the user connected from
9. The local node address where the command was executed
10. Command
11. Command arguments
12. Command body

The ‘isi_audit_viewer’ utility automatically reads the ‘config’ log topic by default, but can also be used read the ‘protocol’ log topic too. Its CLI command syntax is as follows:

# isi_audit_viewer -h

Usage: isi_audit_viewer [ -n <nodeid> | -t <topic> | -s <starttime>|

         -e <endtime> | -v ]

         -n <nodeid> : Specify node id to browse (default: local node)

         -t <topic>  : Choose topic to browse.

            Topics are "config" and "protocol" (default: "config")

         -s <start>  : Browse audit logs starting at <starttime>

         -e <end>    : Browse audit logs ending at <endtime>

         -v verbose  : Prints out start / end time range before printing

             records

Note that, on large clusters where there is heavy (up to the 100,000’s) of audit writes, when running the isi_audit_viewer utility across the cluster with ‘isi_for_array’, it can potentially lead to memory starvation and other issues – especially if outputting to a directory under /ifs. As such, consider directing the output to a non-IFS location such as /var/temp. Also, the isi_audit_viewer ‘-s’ (start time) and ‘-e’ (end time) flags can be used to limit a search (for  1-5 minutes), helping reduce the size of data.

In addition to reading audit events, the view is also a useful tool to assist with troubleshoot any auditing issues. Additionally, any errors that are encountered while processing audit events, and when delivering them to an external CEE server, are written to the log file ‘/var/log/isi_audit_cee.log’. Additionally, the protocol specific logs will contain any issues the audit filter has collecting while auditing events.

The OneFS isi_gather_info utility is the ubiquitous method for collecting and uploading a PowerScale cluster’s context and configuration to assist with the identification and resolution of bugs and issues. As such, it performs the following roles:

• Executes many commands, scripts, and utilities on a cluster, and saves their results
• Collates, or gathers, all these files into a single ‘gzipped’ package
• Optionally transmits this log gather package back to Dell using a choice of several transport methods

By default, a log gather tarfile is written to the /ifs/data/Isilon_Support/pkg/ directory. It can also be uploaded to Dell by the following means:

As indicated in this table, OneFS 9.5 and later releases now leverage FTPS as the default option for FTP upload, thereby protecting the upload of cluster configuration and logs with an encrypted transmission session.

Under the hood, the log gather process comprises an eight phase workflow, with transmission comprising the penultimate ‘Upload’ phase:

The details of each phase are as follows:

Because the isi_gather_info tool is primarily intended for troubleshooting clusters with issues, it runs as root (or compadmin in compliance mode), because it needs to be able to execute under degraded conditions (such as without GMP, during upgrade, and under cluster splits, and so on). Given these atypical requirements, isi_gather_info is built as a standalone utility, rather than using the platform API for data collection.

While FTPS is the new default and recommended transport, the legacy plaintext FTP upload method is still available in OneFS 9.5 and later. As such, Dell’s log server, ftp.isilon.com, also supports both encrypted FTPS and plaintext FTP, so will not impact older release FTP log upload behavior.

This OneFS 9.5 FTPS security enhancement encompasses three primary areas where an FTPS option is now supported:

• Directly executing the /usr/bin/isi_gather_info utility
• Running using the isi diagnostics gather CLI command set
• Creating a diagnostics gather through the OneFS WebUI

For the isi_gather_info utility, two new options are included in OneFS 9.5 and later releases:

Similarly, there are two corresponding options in OneFS 9.5 and later for the isi diagnostics CLI command:

Based on these options, the following table provides some command syntax usage examples, for both FTPS and FTP uploads:

Note that OneFS 9.5 and later releases provide a warning if the cluster admin elects to continue using non-secure FTP for the isi_gather_info tool. Specifically, if the --ftp-insecure option is configured, the following message is displayed, informing the user that plaintext FTP upload is being used, and that the connection and data stream will not be encrypted:

# isi_gather_info --ftp-insecure
You are performing plain text FTP logs upload.
This feature is deprecated and will be removed
in a future release. Please consider the possibility
of using FTPS for logs upload. For further information,
please contact PowerScale support
...

In addition to the command line, log gathers can also be configured using the OneFS WebUI by navigating to Cluster management > Diagnostics > Gather settings.

The Edit gather settings page in OneFS 9.5 and later has been updated to reflect FTPS as the default transport method, plus the addition of radio buttons and text boxes to accommodate the new configuration options.

If plaintext FTP upload is configured, the healthcheck command will display a warning that plaintext upload is used and is no longer a recommended option. For example:

For reference, the OneFS 9.5 and later isi_gather_info CLI command syntax includes the following options:

When a logfile gather arrives at Dell, it is automatically unpacked by a support process and analyzed using the logviewer tool.

Author: Nick Trimbee

As a security best practice, a quarterly security review is recommended. Forming an aggressive security posture for a PowerScale cluster is composed of different facets that may not be applicable to every organization. An organization’s industry, clients, business, and IT administrative requirements determine what is applicable. To ensure an aggressive security posture for a PowerScale cluster, use the checklist in the following table as a baseline for security.

This table serves as a security baseline and must be adapted to specific organizational requirements. See the Dell PowerScale OneFS: Security Considerations | Dell Technologies Info Hub white paper for a comprehensive explanation of the concepts in the table below.

Further, cluster security is not a single event. It is an ongoing process: Monitor this blog for updates. As new updates become available, this post will be updated. Consider implementing an organizational security review on a quarterly basis.

The items listed in the following checklist are not in order of importance or hierarchy but rather form an aggressive security posture as more features are implemented.

Author: Aqib Kazi – Senior Principal Engineering Technologist

In the POSIX world, files typically possess three fundamental timestamps:

These timestamps can be easily viewed from a variety of file system tools and utilities. For example, in this case running ‘stat’ from the OneFS CLI:

# stat -x tstr

  File: "tstr"

  Size: 0            FileType: Regular File

  Mode: (0600/-rw-------)         Uid: (    0/     root)  Gid: (    0/    wheel)

Device: 18446744073709551611,18446744072690335895   Inode: 5103485107    Links: 1

Access: Mon Sep 11 23:12:47 2023

Modify: Mon Sep 11 23:12:47 2023

Change: Mon Sep 11 23:12:47 2023

A typical instance of a change, or “ctime”, timestamp update occurs when a file’s access permissions are altered. Since modifying the permissions doesn’t physically open the file (ie. access the file’s data), its “atime” field is not updated. Similarly, since no modification is made to the file’s contents the “mtime” also remains unchanged. However, the file’s metadata has been changed, and the ctime field is used to record this event. As such, the “ctime” stamp allows a workflow such as a backup application to know to make a fresh copy of the file, including its updated permission values. Similarly, a file rename is another operation that modifies its “ctime” entry without affecting the other timestamps.

Certain other file systems also include a fourth timestamp: namely the “birthtime” of when the file was created. Birthtime (by definition) should never change. It’s also an attribute which organizations and their storage administrators may or may not care about.

Within the Windows file system realm, this “birthtime” timestamp, is affectionally known as “create date”. The create date of a file is essentially the date and time when its inode is “born”.

Note: that this is not a recognized POSIX attribute, like ctime or mtime, rather it is something that was introduced as part of Windows compatibility requirements. And, because it’s a birthtime, linking operations do not necessarily affect it unless a new inode is not created.

As shown below, this create, or birth, date can differ from a file’s modified or accessed dates because the creation date is when that file’s inode version originated. So, for instance, if a file is copied, the new file’s create date will be set to the current time since it has a new inode. This can be seen in the following example where a file is copied from a flash drive mounted on a Windows client’s file system under drive “E:”, to a cluster’s SMB share mounted at drive “Z:”.

The “Date created” date above is ahead in time of both the “accessed” and “modified”, because the latter two were merely inherited from the source file, whereas the create date was set when the copy was made.

The corresponding “date”, “stat”, and “isi get” CLI output from the cluster confirms this:

# stat TEST.txt

18446744072690400151 5103485107 -rw------- 1 root wheel 18446744073709551615 0 "Sep 11 23:12:47 2023" "Sep 11 23:12:47 2023" "Sep 11 23:12:47 2023" "Sep 11 23:12:47 2023" 8192 48 0xe0 tstr

# isi get -Dd TEST.txt

POLICY   W   LEVEL PERFORMANCE   COAL  ENCODING       FILE              IADDRS

default      16+2/2 concurrency   on    UTF-8         tstr              <34,12,58813849600:8192>, <35,3,58981457920:8192>, <69,12,57897025536:8192> ct: 1694473967 rt: 0

*************************************************

* IFS inode: [ 34,12,58813849600:8192, 35,3,58981457920:8192, 69,12,57897025536:8192 ]

*************************************************

*

*  Inode Version:      8

*  Dir Version:        2

*  Inode Revision:     1

*  Inode Mirror Count: 3

*  Recovered Flag:     0

*  Restripe State:     0

*  Link Count:         1

*  Size:               0

*  Mode:               0100600

*  Flags:              0xe0

*  SmartLinked:        False

*  Physical Blocks:    0

*  Phys. Data Blocks:  0

*  Protection Blocks:  0

*  LIN:                1:3031:00b3

*  Logical Size:       0

*  Shadow refs:        0

*  Do not dedupe:      0

*  In CST stats:       False

*  Last Modified:      1694473967.071973000

*  Last Inode Change:  1694473967.071973000

*  Create Time:        1694473967.071973000

*  Rename Time:        0

<snip>

In releases before OneFS 9.5, when a file is replicated, its create date is timestamped when that file was copied from the source cluster. This means when the replication job ran, or, more specifically, when the individual job worker thread got around to processing that specific file.

By way of contrast, OneFS 9.5 and later releases ensure that SyncIQ fully replicates the full array of metadata, preserving all values, including that of the birth time / create date.

The primary consideration for the new create date functionality is that it requires both source and target clusters in a replication set to be running OneFS 9.5 or later.

If either the source or the target is running pre-9.5 code, this time field retains its old behavior of being set to the time of replication (actual file creation) rather than the correct value associated with the source file.

In OneFS 9.5 and later releases, create date timestamping works exactly the same way as SyncIQ replication of other metadata (such as “mtime”, etc), occurring automatically as part of every file replication. Plus, no additional configuration is necessary beyond upgrading both clusters to OneFS 9.5 or later.

One other significant thing to note about this feature is that SyncIQ is changelist-based, using OneFS snapshots under the hood for its checkpointing and delta comparisons.. This means that, if a replication relationship has been configured prior to OneFS 9.5 or later upgrade, the source cluster will have valid birthtime data, but the target’s birthtime data will reflect the local creation time of the files it’s copied.

Note: that, upon upgrading both sides to OneFS 9.5 or later and running a SyncIQ job, nothing will change. This is because SyncIQ will perform its snapshot comparison, determine that no changes were made to the dataset, and so will not perform any replication work. However, if a source file is “touched” so that it’s mtime is changed (or any other action performed that will cause a copy-on-write, or CoW) that will cause the file to show up in the snapshot diff and therefore be replicated. As part of replicating that file, the correct birth time will be written on the target.

Note: that a full replication (re)sync does not get triggered as a result of upgrading a replication cluster pair to OneFS 9.5 or later and thereby enabling this functionality. Instead, any create date timestamp resolution happens opportunistically and in the background as files gets touched or modified - and thereby replicated. Be aware that ‘touching’ a file does change its modification time, in addition to updating the create date, which may be undesirable.

Author: Nick Trimbee

In the data replication world, ensuring your PowerScale clusters' security is paramount. SyncIQ, a powerful data replication tool, requires encryption to prevent unauthorized access.

Concerns about unauthorized replication 

A cluster might inadvertently become the target of numerous replication policies, potentially overwhelming its resources. There’s also the risk of an administrator mistakenly specifying the wrong cluster as the replication target.

Best practices for security 

To secure your PowerScale cluster, Dell recommends enabling SyncIQ encryption as per Dell Security Advisory DSA-2020-039: Dell EMC Isilon OneFS Security Update for a SyncIQ Vulnerability | Dell US. This feature, introduced in OneFS 8.2, prevents man-in-the-middle attacks and addresses other security concerns.

Encryption in new and upgraded clusters 

SyncIQ is disabled by default for new clusters running OneFS 9.1. When SyncIQ is enabled, a global encryption flag requires all SyncIQ policies to be encrypted. This flag is also set for clusters upgraded to OneFS 9.1, unless there’s an existing SyncIQ policy without encryption.

Alternative measures 

For clusters running versions earlier than OneFS 8.2, configuring a SyncIQ pre-shared key (PSK) offers protection against unauthorized replication policies.

By following these security measures, administrators can ensure that their PowerScale clusters are safeguarded against unauthorized access and maintain the integrity and confidentiality of their data.

SyncIQ encryption: securing data in transit

Securing information as it moves between systems is paramount in the data-driven world. Dell PowerScale OneFS release 8.2 has brought a game-changing feature to the table: end-to-end encryption for SyncIQ data replication. This ensures that data is not only protected while at rest but also as it traverses the network between clusters.

Why encryption matters 

Data breaches can be catastrophic, and because data replication involves moving large volumes of sensitive information, encryption acts as a critical shield. With SyncIQ’s encryption, organizations can enforce a global setting that mandates encryption across all SyncIQ policies, to add an extra layer of security.

Test before you implement

It’s crucial to test SyncIQ encryption in a lab environment before deploying it in production. Although encryption introduces minimal overhead, its impact on workflow can vary based on several factors, such as network bandwidth and cluster resources.

Technical underpinnings 

SyncIQ encryption is powered by X.509 certificates, TLS version 1.2, and OpenSSL version 1.0.2o6. These certificates are meticulously managed within the cluster’s certificate stores, ensuring a robust and secure data replication process.

Remember, this is just the beginning of a comprehensive guide about SyncIQ encryption. Stay tuned for more insights about configuration steps and best practices for securing your data with Dell PowerScale’s innovative solutions.

Configuration

Configuring SyncIQ encryption requires a supported OneFS release, certificates, and finally, the OneFS configuration. Before enabling SyncIQ encryption in production, test it in a lab environment that mimics the production setup. Measure the impact on transmission overhead by considering network bandwidth, cluster resources, workflow, and policy configuration.

Here’s a high level summary of the configuration steps:

1. Ensure compatibility:
   1. Ensure that the source and target clusters are running OneFS 8.2 or later.
   2. Upgrade and commit both clusters to OneFS release 8.2 or later.
      
      
2. Create X.509 certificates:
   1. Create X.509 certificates for the source and target clusters using publicly available tools.
   2. The certificate creation process results in the following components:

• Certificate Authority (CA) certificate
• Source certificate and private key
• Target certificate and private key

Note: Some certificate authorities may not generate the public and private key pairs. In that case, manually generate a Certificate Signing Request (CSR) and obtain signed certificates.

3. Transfer certificates to clusters:

1. Transfer the certificates to each cluster.

4. Activate each certificate as follows:

1. Add the source cluster certificate under Data Protection > SyncIQ > Certificates.
2. Add the target server certificate under Data Protection > SyncIQ > Settings.
3. Add the Certificate Authority under Access > TLS Certificates and select Import Authority.

5. Enforce encryption:

1. Each cluster stores its certificate and its peer’s certificate.
2. The source cluster must store the target cluster’s certificate, and vice versa.
3. Storing the peer’s certificate creates a list of approved clusters for data replication.

By following these steps, you can secure your data in transit between PowerScale clusters using SyncIQ encryption. Remember to customize the certificates and settings according to your specific environment and requirements.

For more detailed information about configuring SyncIQ encryption, see SyncIQ encryption | Dell PowerScale SyncIQ: Architecture, Configuration, and Considerations | Dell Technologies Info Hub.

SyncIQ pre-shared key

A SyncIQ pre-shared key (PSK) is configured solely on the target cluster to restrict policies from source clusters without the PSK.

Use Cases: This is recommended for environments without SyncIQ encryption, such as clusters pre-OneFS 8.2 or due to other factors.

SmartLock Compliance: Not supported by SmartLock Compliance mode clusters; upgrading and configuring SyncIQ encryption is advised.

Policy Update: After updating source cluster policies with the PSK, no further configuration is needed. Use the isi sync policies view command to verify.

Remember, configuring the PSK will cause all replicating jobs to the target cluster to fail, so ensure that all SyncIQ jobs are complete or canceled before proceeding.

For more detailed information about configuring a SyncIQ pre-shared key, see SyncIQ pre-shared key | Dell PowerScale SyncIQ: Architecture, Configuration, and Considerations | Dell Technologies Info Hub.

Resources

• SyncIQ encryption | Dell PowerScale SyncIQ: Architecture, Configuration, and Considerations | Dell Technologies Info Hub 
• SyncIQ pre-shared key | Dell PowerScale SyncIQ: Architecture, Configuration, and Considerations | Dell Technologies Info Hub

Author: Aqib Kazi, Senior Principal Engineering Technologist

It’s launch season here at Dell Technologies, and PowerScale is already scaling up spring with the innovative OneFS 9.8 release which shipped today, 9^th April 2024. This new 9.8 release has something for everyone, introducing PowerScale innovations in cloud, performance, serviceability, and ease of use.

 Figure 1. OneFS 9.8 release features

APEX File Storage for Azure

After the debut of APEX File Storage for AWS last year, OneFS 9.8 amplifies PowerScale’s presence in the public cloud by introducing APEX File Storage for Azure.

Figure 2. OneFS 9.8 APEX File Storage for Azure

In addition to providing the same OneFS software platform on-prem and in the cloud as well as customer-managed for full control, APEX File Storage for Azure in OneFS 9.8 provides linear capacity and performance scaling from four to eighteen SSD nodes and up to 3PB per cluster, making it a solid fit for AI, ML, and analytics applications, as well as traditional file shares and home directories and vertical workloads like M&E, healthcare, life sciences, and financial services.

Figure 3. Dell PowerScale scale-out architecture

PowerScale’s scale-out architecture can be deployed on customer-managed AWS and Azure infrastructure, providing the capacity and performance needed to run a variety of unstructured workflows in the public cloud.

Once in the cloud, existing PowerScale investments can be further leveraged by accessing and orchestrating your data through the platform's multi-protocol access and APIs. 

This includes the common OneFS control plane (CLI, WebUI, and platform API) and the same enterprise features, such as Multi-protocol, SnapshotIQ, SmartQuotas, Identity management, and so on.        

Simplicity and efficiency

OneFS 9.8 SmartThrottling is an automated impact control mechanism for the job engine, allowing the cluster to automatically throttle job resource consumption if it exceeds pre-defined thresholds in order to prioritize client workloads. 

OneFS 9.8 also delivers automatic on-cluster core file analysis, and SmartLog provides an efficient, granular log file gathering and transmission framework. Both of these new features help dramatically accelerate the ease and time to resolution of cluster issues.

Performance

OneFS 9.8 also adds support for Remote Direct Memory Access (RDMA) over NFS 4.1 support for applications and clients. This allows for substantially higher throughput performance – especially in the case of single-connection and read-intensive workloads such as machine learning and generative AI model training – while also reducing both cluster and client CPU utilization and provides the foundation for interoperability with NVIDIA’s GPUDirect.

RDMA over NFSv4.1 in OneFS 9.8 leverages the ROCEv2 network protocol. OneFS CLI and WebUI configuration options include global enablement and IP pool configuration, filtering, and verification of RoCEv2 capable network interfaces. NFS over RDMA is available on all PowerScale platforms containing Mellanox ConnectX network adapters on the front end and with a choice of 25, 40, or 100 Gigabit Ethernet connectivity. The OneFS user interface helps easily identify which of a cluster’s NICs support RDMA.

Under the hood, OneFS 9.8 introduces efficiencies such as lock sharding and parallel thread handling, delivering a substantial performance boost for streaming write-heavy workloads such as generative AI inferencing and model training. Performance scales linearly as compute is increased, keeping GPUs busy and allowing PowerScale to easily support AI and ML workflows both small and large. OneFS 9.8 also includes infrastructure support for future node hardware platform generations.

Multipath Client Driver

The addition of a new Multipath Client Driver helps expand PowerScale’s role in Dell Technologies’ strategic collaboration with NVIDIA, delivering the first and only end-to-end large scale AI system. This is based on the PowerScale F710 platform in conjunction with PowerEdge XE9680 GPU servers and NVIDIA’s Spectrum-X Ethernet switching platform to optimize performance and throughput at scale.

In summary, OneFS 9.8 brings the following new features to the Dell PowerScale ecosystem:

We’ll be taking a deeper look at this new functionality in blog articles over the course of the next few weeks. 

Meanwhile, the new OneFS 9.8 code is available on the Dell Online Support site, both as an upgrade and reimage file, allowing installation and upgrade of this new release.

Author: Nick Trimbee

Overview

PowerScale OneFS 9.8 now brings a new offering in Azure — APEX File Storage for Microsoft Azure! It is a software-defined cloud file storage service that provides high-performance, flexible, secure, and scalable file storage for Microsoft Azure environments. It is also a fully customer managed service that is designed to meet the needs of enterprise-scale file workloads running on Azure. This offer joins another native cloud solution that was released last year - APEX File Storage for AWS, for more information, refer to the link: https://www.dell.com/en-us/dt/apex/storage/public-cloud/file.htm?hve=explore+file

Benefits of running OneFS in Cloud

APEX File Storage for Microsoft Azure brings the OneFS distributed file system software into the public cloud, allowing users to have the same management experience in the cloud as with their on-premises PowerScale appliance.

With APEX File Storage for Microsoft Azure, you can easily deploy and manage file storage on Azure. The service provides a scalable and elastic storage infrastructure that can grow, according to your actual business needs.

Some of the key features and benefits of APEX File Storage for Microsoft Azure include:

• Scale-out: APEX File Storage for Microsoft Azure is powered by the Dell PowerScale OneFS distributed file system. You can start with a small OneFS cluster (minimal 4 nodes) and then expand it incrementally as your data storage requirements grow up to 5.6 PiB in cluster capacity with a single namespace (maximum 18 nodes). This large capacity helps support the most demanding, data intensive workloads such as AI.
• Data management: APEX File Storage for Microsoft Azure provides powerful data management capabilities such as: snapshot, data replication, and backup and restore. Because OneFS features are the same in the cloud as they are in on-premises, organizations can simplify operations and reduce management complexity with a consistent user experience.
• Simplified journey to hybrid cloud: More and more organizations operate in a hybrid cloud environment, where they need to move data between on-premises and cloud-based environments. APEX File Storage for Microsoft Azure can help you bridge this gap by facilitating seamless data mobility between on-premises and the cloud with native replication and by providing a consistent data management platform across both environments. Once in the cloud, customers can take advantage of enterprise-class OneFS features such as: multi-protocol support, CloudPools, data reduction, security, and snapshots, to run their workloads in the same way as they do on-premises.
• Data resilience: Ensuring data resilience is critical for businesses to maintain continuity and to safeguard information. APEX File Storage for Microsoft Azure implements erasure coding techniques. This advanced approach optimizes storage efficiency and enhances fault tolerance, enabling the cluster to withstand multiple node failures. By spreading nodes across different racks using Azure availability set, the cluster ensures that data accessibility is maintained in the event of a rack failure.
• High performance: APEX File Storage for Microsoft Azure delivers high-performance file storage with low-latency access to data, ensuring that you can access data quickly and efficiently. Compared to Azure NetApp Files, Dell APEX File Storage for Microsoft Azure enables: about 6x greater cluster performance, up to 11x larger namespace, up to 23x more snapshots per volume, 2x higher cluster resiliency, and an easier and more robust cluster expansion.
• Proactive support: With a 97% customer satisfaction rate, Dell Support Services provides highly trained experts around the clock and around the globe to address your OneFS needs, minimize disruptions, and help you maintain a high level of productivity and outcomes.  

Architecture

APEX File Storage for Microsoft Azure is a software-defined cloud file storage service that combines the power of OneFS distributed file system with the flexibility and scalability of cloud infrastructure. It is a fully customer-managed service that is designed to meet the needs of enterprise-scale file workloads running on Azure.

The architecture of APEX File Storage for Microsoft Azure is built on the OneFS distributed file system. This architecture uses multiple cluster nodes to establish a single global namespace. Each cluster node operates as an instance of the OneFS software, running on an Azure VM to deliver storage capacity and compute resources. It is worth noting that the network bandwidth limit at the Azure VM level is shared between the cluster internal network and the external network.

APEX File Storage for Microsoft Azure uses cloud-native technologies and leverages the elasticity of cloud infrastructure, so that you can easily scale the storage infrastructure as your business requirements grow. APEX File Storage for Microsoft Azure can dynamically scale storage capacity and performance to meet changing demands. It is able to add additional cluster nodes without disruption enabling the storage infrastructure to scale in a more cost-effective and efficient manner. To guarantee the durability and resiliency of data, APEX File Storage for Microsoft Azure distributes data across multiple nodes within the cluster. It also uses advanced data protection techniques such as erasure coding and it provides features such as SyncIQ to ensure that data is available. Even in the event of one or more node failures, the data remains accessible from the remaining cluster nodes.

Availability set and proximity placement group: APEX File Storage for Microsoft Azure is designed to run in an availability set, and the availability set is associated with a dedicated proximity placement group. In this way, APEX File Storage for Microsoft Azure can have better reliability by ensuring more consistent, lower latency on the cluster backend network.

Virtual network: APEX File Storage for Microsoft Azure requires an Azure virtual network to provide network connectivity.

• OneFS cluster internal subnet: The cluster nodes communicate with each other through the internal subnet. The internal subnet must be isolated from VMs that are not in the cluster. Therefore, a dedicated subnet is required for the internal network interfaces of cluster nodes that do not share the internal subnets with other Azure VMs.
• OneFS cluster external subnet: The cluster nodes communicate with clients through the external subnet by using different protocols, such as NFS, SMB, and S3. 
• OneFS cluster internal network interfaces: Network interfaces are in the internal subnet.
• OneFS cluster external network interfaces: Network interfaces are in the external subnet.
• Network security group: The network security group applies to the cluster network interfaces, which allows/denies specific traffic to OneFS cluster.
• Azure VMs: These VMs serve as cluster nodes running the OneFS file system, backed by Azure managed disks. Each node within the cluster is strategically placed in an availability set and a proximity placement group. This configuration ensures that all nodes reside in separate fault domains, enhancing reliability, and it brings them physically closer together to enable lower network latency between cluster nodes. See the Azure availability sets overview and Azure proximity placement groups documentation for more details.

Overall, APEX File Storage for Microsoft Azure offers a powerful and flexible scale-out file storage solution that can help you improve data management, optimize costs, scalability, and security in a cloud-based environment.

Supported cluster configurations

Table 1 shows the supported configuration for APEX File Storage on Azure. It provides you the flexibility to choose different cluster size, various Azure VM size/SKU and so many Azure disk options to meeting your business requirements. For the detailed explanation of these configurations, refer to https://infohub.delltechnologies.com/en-US/t/apex-file-storage-for-microsoft-azure-deployment-guide.

1. Supported configuration for a single cluster

Support Regions

APEX File Storage for Microsoft Azure is globally available. For the detailed regions, refer to https://infohub.delltechnologies.com/en-US/t/apex-file-storage-for-microsoft-azure-deployment-guide.

Performance

Compared to Azure NetApp Files, Dell APEX File Storage for Microsoft Azure enables about 6x greater cluster performance, up to 11x larger namespace, up to 23x more snapshots per volume, 2x higher cluster resiliency, and easier and more robust cluster expansion.

Besides that, I will show you an example of how sequential read and sequential write performance can be linearly scaled out from 4 nodes to 18 nodes to make sure it meets your business requirements. 

This is what we have set up: 

We are using Azure Standard D48ds_v5 VM type and we scale from 10 nodes to 14 nodes and in the end to 18 nodes for testing purposes. With each deployment we kept all the other factors the same, we maintained 12 P40 Azure premium SSDs for data disks in each node. The following table displays the configuration:

The diagram below demonstrates how read performance increases when we scale out our APEX File Storage for Microsoft Azure. You can see a clear positive trend starting from 10 nodes to 18 nodes. The same conclusion also applies with write performance. 

Another example is that you can also scale-up the overall performance of an APEX File Storage for Microsoft Azure by choosing a more powerful Azure VM size/SKU:

In this example, we tested the following Azure VM size/SKU with the same node number (4) and disk number per node (12):

• D32ds_v5
• D48ds_v5
• D64ds_v5
• E104ids_v5

From the results, we can easily find that with the scale-up of Azure VM size/SKU, both read and write performance increase:

 For more details on the performance results and best practices, refer to the following whitepaper https://infohub.delltechnologies.com/en-US/t/introduction-to-apex-file-storage-for-azure-1/.

Resources

https://infohub.delltechnologies.com/en-US/t/apex-file-storage-for-microsoft-azure-deployment-guide

https://infohub.delltechnologies.com/en-US/t/introduction-to-apex-file-storage-for-azure-1/

https://www.dell.com/en-us/blog/ai-anywhere-with-apex-file-storage-for-microsoft-azure/

Authors:

Vincent Shen, Lieven Lin, and Jason He

The surge in artificial intelligence (AI) and machine learning (ML) technologies has sparked a revolution across industries, pushing the boundaries of what's possible. However, this innovation comes with its own set of challenges, particularly when it comes to storage. The heart of AI's potential lies in its ability to process and learn from vast amounts of data, most of which is unstructured. This has placed unprecedented demands on storage solutions, becoming a critical bottleneck for advancing AI technologies.

Navigating the complex landscape of unstructured data storage is no small feat. Traditional storage systems struggle to keep up with the scale and flexibility required by AI workloads. Enterprises find themselves at a crossroads, seeking solutions that can provide scalable, affordable, and fault-tolerant storage. The quest for such a platform is not just about meeting current needs but also paving the way for the future of AI-driven innovation.

The current state of ML and AI

The evolution of ML and AI technologies has reshaped industries far and wide, setting new expectations for data processing and analysis capabilities. These advancements are directly tied to an organization's capacity to handle vast volumes of unstructured data, a domain where traditional storage solutions are being outpaced.

ML and AI applications demand unprecedented levels of data ingestion and computational power, necessitating scalable and flexible storage solutions. Traditional storage systems—while useful for conventional data storage needs—grapple with scalability issues, particularly when faced with the immense file quantities AI and ML workloads generate.

Although traditional object storage methods are capable of managing data as objects within a pool, they fall short when meeting the agility and accessibility requirements essential for AI and ML processes. These storage models struggle with scalability and facilitating the rapid access and processing of data crucial for deep learning and AI algorithms.

The dire necessity of a new kind of storage solution is evident as the current infrastructure is unable to cope with the silos of unstructured data. These silos make it challenging to access, process, and unify data sources, which in turn cripples the effectiveness of AI and ML projects. Furthermore, the maximum storage capacity of traditional storage, tethering at tens of terabytes, is insufficient for the needs of AI-driven initiatives which often require petabytes of data to train sophisticated models.

As ML and AI continue to advance, the quest for a storage solution that can support the growing demands of these technologies remains pivotal. The industry is in dire need of systems that provide ample storage and ensure the flexibility, reliability, and performance efficiency necessary to propel AI and ML into their next phase of innovation.

Understanding unstructured storage demands for AI

The advent of AI and ML has brought unprecedented advancements across industries, enhancing efficiency, accuracy, and the ability to manage and process large datasets. However, the core of these technologies relies on the capability to store, access, and analyze unstructured data efficiently. Understanding the storage demands essential for AI applications is crucial for businesses looking to harness the full power of AI technology.

High throughput and low latency

For AI and ML applications, time is of the essence. The ability to process data at high speeds with high throughput and access it with minimal delay and low latency are non-negotiable requirements. These applications often involve complex computations performed on vast datasets, necessitating quick access to data to maintain a seamless process. For instance, in real-time AI applications such as voice recognition or instant fraud detection, any delay in data processing can critically impact performance and accuracy. Therefore, storage solutions must be designed to accommodate these needs, delivering data as swiftly as possible to the application layer.

Scalability and flexibility

As AI models evolve and the volume of data increases, the need for scalability in storage solutions becomes paramount. The storage architecture must accommodate growth without compromising on performance or efficiency. This is where the flexibility of the storage solutions comes into play. An ideal storage system for AI would scale in capacity and performance, adapting to the changing demands of AI applications over time. Combining the best of on-premises and cloud storage, hybrid storage solutions offer a viable path to achieving this scalability and flexibility. They enable businesses to leverage the high performance of on-premise solutions and the scalability and cost-efficiency of cloud storage, ensuring the storage infrastructure can grow with the AI application needs.

Data durability and availability

Ensuring the durability and availability of data is critical for AI systems. Data is the backbone of any AI application, and its loss or unavailability can lead to significant setbacks in development and performance. Storage solutions must, therefore, provide robust data protection mechanisms and redundancies to safeguard against data loss. Additionally, high availability is essential to ensure that data is always accessible when needed, particularly for AI applications that require continuous operation. Implementing a storage system with built-in redundancy, failover capabilities, and disaster recovery plans is essential to maintain continuous data availability and integrity.

In the context of AI where data is continually ingested, processed, and analyzed, the demands on storage solutions are unique and challenging. Key considerations include maintaining high throughput and low latency for real-time processing, establishing scalability and flexibility to adapt to growing data volumes, and ensuring data durability and availability to support continuous operation. Addressing these demands is critical for businesses aiming to leverage AI technologies effectively, paving the way for innovation and success in the digital era.

What needs to be stored for AI?

The evolution of AI and its underlying models depends significantly on various types of data and artifacts generated and used throughout its lifecycle. Understanding what needs to be stored is crucial for ensuring the efficiency and effectiveness of AI applications.

Raw data

Raw data forms the foundation of AI training. It's the unmodified, unprocessed information gathered from diverse sources. For AI models, this data can be in the form of text, images, audio, video, or sensor readings. Storing vast amounts of raw data is essential as it provides the primary material for model training and the initial step toward generating actionable insights.

Preprocessed data

Once raw data is collected, it undergoes preprocessing to transform it into a more suitable format for training AI models. This process includes cleaning, normalization, and transformation. As a refined version of raw data, preprocessed data needs to be stored efficiently to streamline further processing steps, saving time and computational resources.

Training datasets

Training datasets are a selection of preprocessed data used to teach AI models how to make predictions or perform tasks. These datasets must be diverse and comprehensive, representing real-world scenarios accurately. Storing these datasets allows AI models to learn and adapt to the complexities of the tasks they are designed to perform.

Validation and test datasets

Validation and test datasets are critical for evaluating an AI model's performance. These datasets are separate from the training data and are used to tune the model's parameters and test its generalizability to new, unseen data. Proper storage of these datasets ensures that models are both accurate and reliable.

Model parameters and weights

An AI model learns to make decisions through its parameters and weights. These elements are fine-tuned during training and crucial for the model's decision-making processes. Storing these parameters and weights allows models to be reused, updated, or refined without retraining from scratch.

Model architecture

The architecture of an AI model defines its structure, including the arrangement of layers and the connections between them. Storing the model architecture is essential for understanding how the model processes data and for replicating or scaling the model in future projects.

Hyperparameters

Hyperparameters are the configuration settings used to optimize model performance. Unlike parameters, hyperparameters are not learned from the data but set prior to the training process. Storing hyperparameter values is necessary for model replication and comparison of model performance across different configurations.

Feature engineering artifacts

Feature engineering involves creating new input features from the existing data to improve model performance. The artifacts from this process, including the newly created features and the logic used to generate them, need to be stored. This ensures consistency and reproducibility in model training and deployment.

Results and metrics

The results and metrics obtained from model training, validation, and testing provide insights into model performance and effectiveness. Storing these results allows for continuous monitoring, comparison, and improvement of AI models over time.

Inference data

Inference data refers to new, unseen data that the model processes to make predictions or decisions after training. Storing inference data is key for analyzing the model's real-world application and performance and making necessary adjustments based on feedback.

Embeddings

Embeddings are dense representations of high-dimensional data in lower-dimensional spaces. They play a crucial role in processing textual data, images, and more. Storing embeddings allows for more efficient computation and retrieval of similar items, enhancing model performance in recommendation systems and natural language processing tasks.

Code and scripts

The code and scripts used to create, train, and deploy AI models are essential for understanding and replicating the entire AI process. Storing this information ensures that models can be retrained, refined, or debugged as necessary.

Documentation and metadata

Documentation and metadata provide context, guidelines, and specifics about the AI model, including its purpose, design decisions, and operating conditions. Proper storage of this information supports ethical AI practices, model interpretability, and compliance with regulatory standards.

Challenges of unstructured data in AI

In the realm of AI, handling unstructured data presents a unique set of challenges that must be navigated carefully to harness its full potential. As AI systems strive to mimic human understanding, they face the intricate task of processing and deriving meaningful insights from data that lacks a predefined format. This section delves into the core challenges associated with unstructured data in AI, primarily focusing on data variety, volume, and velocity.

Data variety

Data variety refers to the myriad types of unstructured data that AI systems are expected to process, ranging from texts and emails to images, videos, and audio files. Each data type possesses its unique characteristics and demands specific preprocessing techniques to be effectively analyzed by AI models.

• Richer Insights but Complicated Processing: While the diverse data types can provide richer insights and enhance model accuracy, they significantly complicate the data preprocessing phase. AI tools must be equipped with sophisticated algorithms to identify, interpret, and normalize various data formats.
• Innovative AI Applications: The advantage of mastering data variety lies in the development of innovative AI applications. By handling unstructured data from different domains, AI can contribute to advancements in natural language processing, computer vision, and beyond.

Data volume

The sheer volume of unstructured data generated daily is staggering. As digital interactions increase, so does the amount of data that AI systems need to analyze.

• Scalability Challenges: The exponential growth in data volume poses scalability challenges for AI systems. Storage solutions must not only accommodate current data needs but also be flexible enough to scale with future demands.
• Efficient Data Processing: AI must leverage parallel processing and cloud storage options to keep up with the volume. Systems designed for high-throughput data analysis enable quicker insights, which are essential for timely decision-making and maintaining relevance in a rapidly evolving digital landscape.

Data velocity

Data velocity refers to the speed at which new data is generated and the pace at which it needs to be processed to remain actionable. In the age of real-time analytics and instant customer feedback, high data velocity is both an opportunity and a challenge for AI.

• Real-Time Processing Needs: AI systems are increasingly required to process information in real-time or near-real-time to provide timely insights. This necessitates robust computational infrastructure and efficient data streaming technologies.
• Constant Adaptation: The dynamic nature of unstructured data, coupled with its high velocity, demands that AI systems constantly adapt and learn from new information. Maintaining accuracy and relevance in fast-moving data environments is critical for effective AI performance.

In addressing these challenges, AI and ML technologies are continually evolving, developing more sophisticated systems capable of handling the complexity of unstructured data. The key to unlocking the value hidden within this data lies in innovative approaches to data management where flexibility, scalability, and speed are paramount.

Strategies to manage unstructured data in AI

The explosion of unstructured data poses unique challenges for AI applications. Organizations must adopt effective data management strategies to harness the full potential of AI technologies. In this section, we delve into key strategies like data classification and tagging and the use of PowerScale clusters to efficiently manage unstructured data in AI.

Data classification and tagging

Data classification and tagging are foundational steps in organizing unstructured data and making it more accessible for AI applications. This process involves identifying the content and context of data and assigning relevant tags or labels, which is crucial for enhancing data discoverability and usability in AI systems.

• Automated tagging tools can significantly reduce the manual effort required to label data, employing AI algorithms to understand the content and context automatically.
• Custom metadata tags allow for the creation of a rich set of file classification information. This not only aids in the classification phase but also simplifies later iterations and workflow automation.
• Effective data classification enhances data security by accurately categorizing sensitive or regulated information, enabling compliance with data protection regulations.

Implementing these strategies for managing unstructured data prepares organizations for the challenges of today's data landscape and positions them to capitalize on the opportunities presented by AI technologies. By prioritizing data classification and leveraging solutions like PowerScale clusters, businesses can build a strong foundation for AI-driven innovation.

Best practices for implementing AI storage solutions

Implementing the right AI storage solutions is crucial for businesses seeking to harness the power of artificial intelligence. With the explosive growth of unstructured data, adhering to best practices that optimize performance, scalability, and cost is imperative. This section delves into key practices to ensure your AI storage infrastructure meets the demands of modern AI workloads.

Assess workload requirements

Before diving into storage solutions, one must thoroughly assess AI workload requirements. Understanding the specific needs of your AI applications—such as the volume of data, the necessity for high throughput/low latency, and the scalability and availability requirements—is fundamental. This step ensures you select the most suitable storage solution that meets your application's needs.

AI workloads are diverse, with each having unique demands on storage infrastructure. For instance, training a machine learning model could require rapid access to vast amounts of data, whereas inference workloads may prioritize low latency. An accurate assessment leads to an optimized infrastructure, ensuring that storage solutions are neither overprovisioned nor underperforming, thereby supporting AI applications efficiently and cost-effectively.

Leverage PowerScale

For managing large volumes and varieties of unstructured data, leveraging PowerScale nodes offers a scalable and efficient solution. PowerScale nodes are designed to handle the complexities of AI and machine learning workloads, offering optimized performance, scalability, and data mobility. These clusters allow organizations to store and process vast amounts of data efficiently for a range of AI use cases due to the following:

• Scalability is a key feature, with PowerScale clusters capable of growing with the organization's data needs. They support massive capacities, allowing businesses to store petabytes of data seamlessly.
• Performance is optimized for the demanding workloads of AI applications with the ability to process large volumes of data at high speeds, reducing the time for data analyses and model training.
• Data mobility within PowerScale clusters on-premise and in the cloud ensures that data can be accessed when and where needed, supporting various AI and machine learning use cases across different environments.

PowerScale clusters allow businesses to start small and grow capacity as needed, ensuring that storage infrastructure can scale alongside AI initiatives without compromising on performance. The ability to handle multiple data types and protocols within a single storage infrastructure simplifies management and reduces operational costs, making PowerScale nodes an ideal choice for dynamic AI environments.

Utilize PowerScale OneFS 9.7.0.0

PowerScale OneFS 9.7.0.0 is the latest version of  the Dell PowerScale operating system for scale-out network-attached storage (NAS). OneFS 9.7.0.0 introduces several enhancements in data security, performance, cloud integration, and usability. 

OneFS 9.7.0.0 extends and simplifies the PowerScale offering in the public cloud, providing more features across various instance types and regions. Some of the key features in OneFS 9.7.0.0 include:

• Cloud Innovations: Extends cloud capabilities and features, building upon the debut of APEX File Storage for AWS
• Performance Enhancements: Enhancements to overall system performance
• Security Enhancements: Enhancements to data security features
• Usability Improvements: Enhancements to make managing and using PowerScale easier

Employ PowerScale F210 and F710

PowerScale, through its continuous innovation, extends into the AI era by introducing the next generation of PowerEdge-based nodes: the PowerScale F210 and F710. These new all-flash nodes leverage the Dell PowerEdge R660 from the PowerEdge platform, unlocking enhanced performance capabilities.

On the software front, both the F210 and F710 nodes benefit from significant performance improvements in PowerScale OneFS 9.7. These nodes effectively address the most demanding workloads by combining hardware and software innovations. The PowerScale F210 and F710 nodes represent a powerful combination of hardware and software advancements, making them well-suited for a wide range of workloads. For more information on the F210 and F710, see PowerScale All-Flash F210 and F710 | Dell Technologies Info Hub.

Ensure data security and compliance

Given the sensitivity of the data used in AI applications, robust security measures are paramount. Businesses must implement comprehensive security strategies that include encryption, access controls, and adherence to data protection regulations. Safeguarding data protects sensitive information and reinforces customer trust and corporate reputation.

Compliance with data protection laws and regulations is critical to AI storage solutions. As regulations can vary significantly across regions and industries, understanding and adhering to these requirements is essential to avoid significant fines and legal challenges. By prioritizing data security and compliance, organizations can mitigate risks associated with data breaches and non-compliance.

Monitor and optimize

Continuous storage environment monitoring and optimization are essential for maintaining high performance and efficiency. Monitoring tools can provide insights into usage patterns, performance bottlenecks, and potential security threats, enabling proactive management of the storage infrastructure.

Regular optimization efforts can help fine-tune storage performance, ensuring that the infrastructure remains aligned with the evolving needs of AI applications. Optimization might involve adjusting storage policies, reallocating resources, or upgrading hardware to improve efficiency, reduce costs, and ensure that storage solutions continue to effectively meet the demands of AI workloads.

By following these best practices, businesses can build and maintain a storage infrastructure that supports their current AI applications and is poised for future growth and innovation.

Conclusion

Navigating the complexities of unstructured storage demands for AI is no small feat. Yet, by adhering to the outlined best practices, businesses stand to benefit greatly. The foundational steps include assessing workload requirements, selecting the right storage solutions, and implementing robust security measures. Furthermore, integrating PowerScale nodes and a commitment to continuous monitoring and optimization are key to sustaining high performance and efficiency. As the landscape of AI continues to evolve, these practices will not only support current applications but also pave the way for future growth and innovation. In the dynamic world of AI, staying ahead means being prepared, and these strategies offer a roadmap to success.

Frequently asked questions

How big are AI data centers?

Data centers catering to AI, such as those by Amazon and Google, are immense, comparable to the scale of football stadiums.

How does AI process unstructured data?

AI processes unstructured data including images, documents, audio, video, and text by extracting and organizing information. This transformation turns unstructured data into actionable insights, propelling business process automation and supporting AI applications.

How much storage does an AI need?

AI applications, especially those involving extensive data sets, might require significant memory, potentially as much as 1TB or more. Such vast system memory efficiently facilitates the processing and statistical analysis of entire data sets.

Can AI handle unstructured data?

Yes, AI is capable of managing both structured and unstructured data types from a variety of sources. This flexibility allows AI to analyze and draw insights from an expansive range of data, further enhancing its utility across diverse applications.

Author: Aqib Kazi, Senior Principal Engineer, Technical Marketing

SIA recently unveiled its 2024 Security Megatrend report in which AI prominently claims the top position, dominating all four top spots. With AI making waves across global industries, there arises a set of concerns that demand thoughtful consideration. The key megatrends highlighted are as follows:

• AI: Security of AI
• AI: Visual Intelligence (Distinct from Video Surveillance)
• AI: Generative AI
• AI: Regulations of AI

This discussion will specifically delve into the first two trends—AI Security and Visual Intelligence.

Security of AI

The top spot on the list is occupied by the security of AI. Ironically, the most effective security for AI is AI itself. AI is tasked with monitoring behaviors related to data creation and access, identifying anomalies indicative of potential malicious activities. As businesses increasingly adopt AI, the value of data rises significantly for the organization. However, with AI becoming a more integral operational component, a cyber incident could disrupt not only data but also overall operations and production, particularly when there's a lack of metadata for decision-making.

Ensuring robust cyber protection for data becomes crucial, and solutions like the Ransomware Defender in Dell Technologies' unstructured data offering play a key role. Cyber recovery strategies are also imperative to swiftly resume normal operations. An air-gapped cyber recovery vault is essential, minimizing disruptions and securing a clean and complete dataset for rapid recovery from incidents.

 Figure 1. Air-gapped cyber recovery vault

AI visual intelligence

AI Visual Intelligence has been increasingly used across various industries for a multitude of purposes, including object recognition and classification, anomaly detection, predictive analytics, customer insights and experience enhancement, autonomous systems, healthcare diagnostics, environmental monitoring, and surveillance and security. By integrating AI Visual Intelligence into their operations, businesses can harness the power of visual data to improve decision-making, automate processes, enhance efficiencies, and unlock new opportunities for innovation and growth.

Video extends beyond security to impact business operations, enhancing efficiencies as the metadata collected from cameras serves business use cases beyond security functions. An example is the collection of this metadata, such as image metadata, timestamps, objects metadate, geo location, and more. The collection of this metadata necessitates a robust storage solution to preserve complete datasets readily available for models to achieve desired outcomes. This data is considered a mission-critical workload, demanding optimal uptime for storage solutions.

Adopting an N+X node-based storage architecture on-premises guarantees that data is consistently written and available, providing 99.9999% (6 nines) availability in an on-prem cloud environment. Dell Unstructured Data Solutions align perfectly with this workload, ensuring uninterrupted business operations compared to server-based storage solutions facing challenges during deployment or encountering issues with public cloud connectivity. The potential cost-prohibitive nature of public cloud storage for the data required in regular AI modeling may lead to a continued trend of cloud repatriation to on-premises.

Security practitioners evaluating the need for cameras must now strategically map out potential stakeholders within organizations to determine camera requirements aligned with their business outcomes. This strategic approach is anticipated to drive a higher demand for cameras and associated services.

Resources

Check out Dell PowerScale for more information about Dell PowerScale solutions.

Authors: Mordi Shushan, Brian Stonge  

Generative AI systems thrive on vast amounts of unstructured data, which are essential for training algorithms to recognize patterns, make predictions, and generate new content. Unstructured data – such as text, images, and audio – does not follow a predefined model, making it more complex and varied than structured data.

Preprocessing unstructured data

Unstructured data does not have a predefined format or schema, including text, images, audio, video, or documents. Preprocessing unstructured data involves cleaning, normalizing, and transforming the data into a structured or semi-structured form that the AI can understand and that can be used for analysis or machine learning.

Preprocessing unstructured data for generative AI is a crucial step that involves preparing the raw data for use in training AI models. The goal is to enhance the quality and structure of the data to improve the performance of generative models.

There are different steps and techniques for preprocessing unstructured data, depending on the type and purpose of the data. Some common steps are:

• Data completion: This step involves filling in missing or incomplete data, either by using average or estimated values or by discarding or ignoring the data points with missing fields.
• Data noise reduction: This step involves removing or reducing irrelevant, redundant, or erroneous data, such as duplicates, spelling errors, hidden objects, or background noise.
• Data transformation: This step involves converting the data into a standard or consistent format, including scaling and normalizing numerical data, encoding categorical data, or extracting features from text, image, audio, or video data.
• Data reduction: This step involves reducing the dimensionality or size of the data, either by selecting a subset of relevant features or data points or by applying techniques such as principal component analysis, clustering, or sampling.
• Data validation: This step involves checking the quality and accuracy of the preprocessed data by using statistical methods, visualization tools, or domain knowledge.

These steps can help enhance the quality, reliability, and interpretability of the data, which can improve the performance and outcomes of the analysis or machine learning models.

PowerScale F210 and F710 platform

PowerScale’s continuous innovation extends into the AI era with the introduction of the next generation of PowerEdge-based nodes, including the PowerScale F210 and F710. The new PowerScale all-flash nodes leverage Dell PowerEdge R660, unlocking the next generation of performance. On the software front, the F210 and F710 take advantage of significant performance improvements in PowerScale OneFS 9.7. Combining the hardware and software innovations, the F210 and F710 tackle the most demanding workloads with ease.

The F210 and F710 offer greater density in a 1U platform, with the F710 supporting 10 NVMe SSDs per node and the F210 offering a 15.36 TB drive option. The Sapphire Rapids CPU provide 19% lower cycles-per-instruction. PCIe Gen 5 doubles throughput when compared to PCIe Gen 4. Additionally, the nodes take advantage of DDR5, offering greater speed and bandwidth.

From a software perspective, PowerScale OneFS 9.7 introduces a significant leap in performance. OneFS 9.7 updates the protocol stack, locking, and direct-write. To learn more about OneFS 9.7, check out this article on PowerScale OneFS 9.7.

The OneFS journal in the all-flash F210 and F710 nodes uses a 32 GB configuration of the Dell Software Defined Persistent Memory (SDPM) technology. Previous platforms used NVDIMM-n for persistent memory, which consumed a DIMM slot.

For more details about the F210 and F710, see our other blog post at Dell.com: https://www.dell.com/en-us/blog/next-gen-workloads-require-next-gen-storage/.

Performance

The introduction of the PowerScale F210 and F710 nodes capitalizes on significant leaps in hardware and software from the previous generations. OneFS 9.7 introduces tremendous performance-oriented updates, including the protocol stack, locking, and direct-write. The PowerEdge-based servers offer a substantial hardware leap from previous generations. The hardware and software advancements combine to offer enormous performance gains, particularly for streaming reads and writes.

PowerScale F210

The PowerScale F210 is a 1U chassis based on the PowerEdge R660. A minimum of three nodes is required to form a cluster, with a maximum of 252 nodes. The F210 is node pool compatible with the F200.

Table 1. F210 specifications

PowerScale F710

The PowerScale F710 is a 1U chassis based on the PowerEdge R660. A minimum of three nodes is required to form a cluster, with a maximum of 252 nodes.

Table 2. F710 specifications

For more details on the new PowerScale all-flash platforms, see the PowerScale All-Flash F210 and F710 white paper.

Author: Aqib Kazi

As we know, when users access OneFS cluster data via different protocols, the final permission enforcement happens on the OneFS file system. In OneFS, this is achieved by the Access Control Lists (ACLs) implementation, which provides granular permission control on directories and files. In this article, we will look at the basics of OneFS ACLs.

OneFS ACL

OneFS provides a single namespace for multiprotocol access and has its own internal ACL representation to perform access control. The internal ACL is presented as protocol-specific views of permissions so that NFS exports display POSIX mode bits for NFSv3 and ACL for NFSv4 and SMB. 

When connecting to an PowerScale cluster with SSH, you can manage not only POSIX mode bits but also ACLs with standard UNIX tools such as chmod commands. In addition, you can edit ACL policies through the web administration interface to configure OneFS permissions management for networks that mix Windows and UNIX systems.

The OneFS ACL design is derived from Windows NTFS ACL. As such, many of its concept definitions and operations are similar to the Windows NTFS ACL, such as ACE permissions and inheritance.

OneFS synthetic ACL and real ACL

To deliver cross-protocol file access seamlessly, OneFS stores an internal representation of a file-system object’s permissions. The internal representation can contain information from the POSIX mode bits or the ACL. 

OneFS has two types of ACLs to fulfill different scenarios:

• OneFS synthetic ACL: Under the default ACL policy, if no inheritable ACL entries exist on a parent directory – such as when a file or directory is created through a NFS or SSH session on OneFS within the parent directory – the directory will only contain POSIX mode bits permission. OneFS uses the internal representation to generate a OneFS synthetic ACL, which is an in-memory structure that approximates the POSIX mode bits of a file or directory for an SMB or NFSv4 client. 
• OneFS real ACL: Under the default ACL policy, when a file or directory is created through SMB or when the synthetic ACL of a file or directory is modified through an NFSv4 or SMB client, the OneFS real ACL is initialized and stored on disk. The OneFS real ACL can also be initialized using the OneFS enhanced chmod command tool with the +a, -a, or =a option to modify the ACL. 

OneFS access control entries

In contrast to the Windows DACL and NFSv4 ACL, the OneFS ACL access control entry (ACE) adds an additional identity type. OneFS ACEs contain the following information:

• Identity name: The name of a user or group
• ACE type: The type of the ACE (allow or deny)
• ACE permissions and inheritance flags: A list of permissions and inheritance flags separated with commas

OneFS ACE permissions

Similar to the Windows permission level, OneFS divides permissions into the following three types:

• Standard ACE permissions: These apply to any object in the file system
• Generic ACE permissions: These map to a bundle of specific permissions
• Constant ACE permissions: These are specific permissions for file-system objects

The standard ACE permissions that can appear for a file-system object are shown in the following table:

The generic ACE permissions that can appear for a file system object are shown in the following table:

The constant ACE permissions that can appear for a file-system object are shown in the following table:

OneFS ACL inheritance

Inheritance allows permissions to be layered or overridden as needed in an object hierarchy and allows for simplified permissions management. The semantics of OneFS ACL inheritance are the same as Windows ACL inheritance and will feel familiar to someone versed in Windows NTFS ACL inheritance. The following table shows the ACE inheritance flags defined in OneFS:

Author: Lieven Lin

The Dell PowerScale CloudPools feature of OneFS allows tiering cold or infrequently accessed data to move to lower-cost cloud storage. CloudPools extends the PowerScale namespace to the private cloud, or the public cloud. For CloudPools supported cloud providers, see the CloudPools Supported Cloud Providers blog.

This blog focuses on the following CloudPools operation workflows:

• Archive
• Recall
• Read
• Update

Archive

The archive operation is the CloudPools process of moving file data from the local PowerScale cluster to cloud storage. Files are archived either using the SmartPools Job or from the command line. The CloudPools archive process can be paused or resumed.

The following figure shows the workflow of the CloudPools archive.

Figure 1.  Archive workflow

More workflow details include:

• The file pool policy in Step 1 specifies a cloud target and cloud-specific parameters. Policy examples include:
• Encryption: CloudPools provides an option to encrypt data before the data is sent to the cloud storage. It uses the PowerScale key management module for data encryption and uses AES-256 as the encryption algorithm. The benefit of encryption is that only encrypted data is being sent over the network.
• Compression: CloudPools provides an option to compress data before the data is sent to the cloud storage. It implements block-level compression using the zlib compression library. CloudPools does not compress data that is already compressed.
• Local data cache: Caching is used to support local reading and writing of SmartLink files. To optimize performance, it reduces bandwidth costs by eliminating repeated fetching of file data for repeated reads and writes. The data cache is used for temporarily caching file data from the cloud storage on PowerScale disk storage for files that have been moved off cluster by CloudPools.
• Data retention: Data retention is a concept used to determine how long to keep cloud objects on the cloud storage.
• When chunks are sent from the PowerScale cluster to cloud in Step 3, a checksum is applied for each chunk to ensure data integrity.

Recall

The recall operation is the CloudPools process of reversing the archive process. It replaces the SmartLink file by restoring the original file data on the PowerScale cluster and removing the cloud objects in cloud. The recall process can only be performed using the command line. The CloudPools recall process can be paused or resumed.

The following figure shows the workflow of CloudPools recall. 

Figure 2.  Recall workflow

Read

The read operation is the CloudPools process of client data access, known as inline access. When a client opens a file for read, the blocks are added to the cache in the associated SmartLink file by default. The cache can be disabled by setting the accessibility in the file pool policy for CloudPools. The accessibility setting is used to specify how data is cached in SmartLink files when a user or application accesses a SmartLink file on the PowerScale cluster. Values are cached (default) and no cache.

The following figure shows the workflow of CloudPools read by default. 

Figure 3.  Read workflow

Starting from OneFS 9.1.0.0, cloud object cache is introduced to enhance CloudPools functions for communicating with cloud. In Step 1, OneFS looks for data in the object cache first and OneFS retrieves data from the object cache if the data is already in the object cache. Cloud object cache reduces the number of requests to cloud when reading a file.

Prior to OneFS 9.1.0.0, in Step 1, OneFS looks for data in the local data cache first. It moves to Step 3 if the data is already in the local data cache.

Note: Cloud object cache is per node. Each node maintains its own object cache on the cluster. 

Update

The update operation is the CloudPools process that occurs when clients update data. When clients change to a SmartLink file, CloudPools first writes the changes in the data local cache and then periodically sends the updated file data to cloud. The space used by the cache is temporary and configurable.

The following figure shows the workflow of CloudPools update. 

Figure 4.  Update workflow

Thank you for taking the time to read this blog, and congratulations on gaining a clear understanding of how the OneFS CloudPools operation works!

Author: Jason He, Principal Engineering Technologist

This blog focuses on CloudPools reporting, specifically:

• CloudPools network stats
• The isi_fsa_pools_usage feature

CloudPools network stats

Dell PowerScale CloudPools network stats collect every network transaction and provide network activity statistics from CloudPools connections to the cloud storage.

Displaying network activity statistics

The network activity statistics include bytes In, bytes Out, and the number of GET, PUT, and DELETE operations. CloudPools network stats are available in two categories:

• Per CloudPools account
• Per file pool policy

Note: CloudPools network stats do not provide file statistics, such as the file list being archived or recalled.

Run the following command to check the CloudPools network stats by CloudPools account:

isi_test_cpool_stats -Q --accounts <account_name>

For example, the following command shows the current CloudPools network stats by CloudPools account:

isi_test_cpool_stats -Q --accounts testaccount
Account Name   Bytes In    Bytes Out   Num Reads   Num Writes   Num Deletes
testaccount    4194896000  4194905034  4000        2001         8001  

Similarly, you can run the following command to check the CloudPools network stats by file pool policy:

isi_test_cpool_stats -Q --policies <policy_name>

And here is an example of current CloudPools network stats by file pool policy:

isi_test_cpool_stats -Q --policies testpolicy
Policy Name    Bytes In       Bytes Out      Num Reads      Num Writes
testpolicy     4154896000     4154905034     4000           2001

Note: The command output does not include the number of deletes by file pool policy.

Run the following command to check the history for CloudPools network stats:

isi_test_cpool_stats -q –s <number of seconds in the past to start stat query>

Use the s parameter to define the number of seconds in the past. For example, set it as 86,400 to query CloudPools network stats over the last day, as in the following example:

isi_test_cpool_stats -q -s 86400
Account          bytes-in     bytes-out    gets   puts   deletes
testaccount    | 4194896000 | 4194905034 | 4000 | 2001 | 8001

You can also run the following command to flush stats from memory to database and get the real-time CloudPools network stats:

isi_test_cpool_stats -f

Displaying stats for CloudPools activities

The cloud statistics namespace with CloudPools is added in OneFS 9.4.0.0. This feature leverages existing OneFS daemons and systems to track statistics about CloudPools activities. The statistics include bytes In, bytes Out, and the number of Reads, Writes, and Deletions. CloudPools statistics are available in two categories:

• Per CloudPools account
• Per file pool policy

Note: The cloud statistics namespace with CloudPools does not provide file statistics, such as the file list being archived or recalled.

You can run these isi statistics cloud commands to view statistics about CloudPools activities:

isi statistics cloud --account <account_name>
isi statistics cloud --policy <policy_name>

The following command shows an example of current CloudPools statistics by CloudPools account:

isi statistics cloud --account s3                    
Account Policy In      Out     Reads   Writes  Deletions       Cloud      Node
s3             218.5KB 218.7KB 1       2       0               AWS        3
s3             0.0B    0.0B    0       0       0               AWS        1
s3             0.0B    0.0B    0       0       0               AWS        2

The following command shows an example of current CloudPools statistics by file pool policy:

isi statistics cloud --policy s3policy        
Account Policy         In      Out     Reads   Writes  Deletions  Cloud       Node
s3      s3policy       218.5KB 218.7KB  1      2       0          AWS         3
s3      s3policy       0.0B    0.0B     0      0       0          AWS         1
s3      s3policy       0.0B    0.0B     0      0       0          AWS         2

The isi_fsa_pools_usage feature

Starting from OneFS 8.2.2, you can run the following command to list Logical Size and Physical Size of stubs in one directory. This feature leverages IndexUpdate and FSA (File System Analytics) jobs. To enable this feature, it requires:

• Scheduling the IndexUpdate job. It’s recommended to run it every four hours.
• Scheduling the FSA job. It’s recommended to run it every day, but not more often than the IndexUpdate job.

isi_fsa_pools_usage /ifs
Node Pool                  Dirs  Files  Streams  Logical Size   Physical Size
Cloud                      0     1       0       338.91k           24.00k
h500_30tb_3.2tb-ssd_128gb  42    300671  0       879.23G            1.20T

Now, you get how to use commands for CloudPools reporting. It’s simple and straightforward. Thanks for reading!

Author: Jason He, Principal Engineering Technologist

Dell PowerScale CloudPools SmartLink files are the sole means to access file data stored in the cloud, so ensure that you protect them from accidental deletion.

Note: SmartLink files cannot be backed up using a copy command, such as secure copy (scp).

This blog focuses on backing up SmartLink files using OneFS SyncIQ and NDMP (Network Data Management Protocol).

When the CloudPools version differs between the source cluster and the target PowerScale cluster, the CloudPools cross-version compatibility is handled.

NDMP and SyncIQ provide two types of copy or backup:

• Shallow copy (SC)/backup: Replicates or backs up SmartLink files to the target PowerScale cluster or tape as SmartLink files without file data.
• Deep copy (DC)/backup: Replicates or backs up SmartLink files to the target PowerScale cluster or tape as regular files or unarchived files. The backup or replication will be slower than for a shallow copy backup. Disk space will be consumed on the target cluster for replicating data.

The following table shows the CloudPools and OneFS mapping information. CloudPools 2.0 is released along with OneFS 8.2.0. CloudPools 1.0 is running in OneFS 8.0.x or 8.1.x.

Table 1.  CloudPools and OneFS mapping information

The following table shows the NDMP and SyncIQ supported use cases when different versions of CloudPools are running on the source and target clusters. As noted in the following table, if CloudPools 2.0 is running on the source PowerScale cluster and CloudPools 1.0 is running on the target PowerScale cluster, shallow copies are not allowed.

Table 2.  NDMP and SyncIQ supported use cases with CloudPools  

SyncIQ

SyncIQ is CloudPools-aware but consider the guidance in snapshot efficiency, especially where snapshot retention periods on the target cluster will be long.

SyncIQ policies support two types of data replication for CloudPools:

• Shallow copy: This option is used to replicate files as SmartLink files without file data from the source PowerScale cluster to the target PowerScale cluster.
• Deep copy: This option is used to replicate files as regular files or unarchived files from the source PowerScale cluster to the target PowerScale cluster.

SyncIQ, SmartPools, and CloudPools licenses are required on both the source and target PowerScale cluster. It is highly recommended to set up a scheduled SyncIQ backup of the SmartLink files.

When SyncIQ replicates SmartLink files, it also replicates the local cache state and unsynchronized cache data from the source PowerScale cluster to the target PowerScale cluster. The following figure shows the SyncIQ replication when replicating directories including SmartLink files and unarchived normal files. Both unidirectional and bi-directional replication are supported.

Note: OneFS manages cloud access at the cluster level and does not support managing cloud access at the directory level. When failing over a SyncIQ directory containing SmartLink files to a target cluster, you need to remove cloud access on the source cluster and add cloud access on the target cluster. If there are multiple CloudPools storage accounts, removing/adding cloud access will impact all CloudPools storage accounts on the source/target cluster.

Figure 1.  SyncIQ replication

Note: If encryption is enabled in a file pool policy for CloudPools, SyncIQ also replicates all the relevant encryption keys to the secondary PowerScale cluster along with the SmartLink files.

NDMP

NDMP is also CloudPools-aware and supports three backup and restore methods for CloudPools:

• DeepCopy: This option is used to back up files as regular files or unarchived files. Files can only be restored as regular files.
• ShallowCopy: This option is used to back up files as SmartLink files without file data. Files can only be restored as SmartLink files.
• ComboCopy: This option is used to back up files as SmartLink files with file data. Files can be restored as regular files or SmartLink files.

It is possible to update the file data and send the updated data to the cloud storage. Multiple version SmartLink files can be backed up to tape using NDMP, and multiple versions of CDOs (Cloud Data Objects) are protected in the cloud under the data retention setting. You can restore a specific version of a SmartLink file from tape to a PowerScale cluster and continue to access (read or update) the file as before.

Note: If encryption is enabled in the file pool policy for CloudPools, NDMP also backs up all the relevant encryption keys to tapes along with the SmartLink files.

Thank you for taking the time to read this blog, and congratulations on knowing the solutions for protecting SmartLink files using OneFS SyncIQ and NDMP.

Author: Jason He, Principal Engineering Technologist

When sizing a storage solution for OneFS, two major aspects need to be considered – capacity and performance. In this blog, we will talk about how to calculate the raw capacity in each node in the AWS cloud environment.

Consider a customer who wants to have 30TB of data capacity on APEX File Storage on AWS. The data reduction ratio is 1.6, and the cluster contains 6 nodes. How much capacity is needed for each node of the cluster?

1. The usable capacity is calculated by dividing the application data size by the data reduction ratio: 30TB/1.6 = 18.75TB

2. OneFS in the AWS environment uses +2n as the default protection level. The +2n protection level striping pattern of 6 nodes is 4+2. The raw capacity necessary can be calculated by dividing the usable capacity by the striping pattern for the number of nodes involved: 18.75TB/66% = 28.41TB

3. Single disk capacity is then calculated by dividing the total raw capacity by the number of nodes involved:  28.41TB/6 nodes = 4.735TB

4. When each node contains 10 disks, each disk’s raw capacity should be 474GB.

OK, let's take a look at the formula of this calculation:

For reference, the striping patterns of 4, 5, and 6 nodes are listed as follows:

* 4 nodes: 2+2 (50%)

* 5 nodes: 3+2 (60%)

* 6 nodes: 4+2 (66%)

Now, knowing the logical data capacity, you can calculate the appropriate amount of capacity of each single EBS volume in the cluster.

Author: Yunlong Zhang

Starting with OneFS version 9.0, PowerScale enables data access through the Amazon Simple Storage Service (Amazon S3) application programing interface (API) natively. PowerScale implements the S3 API as a first-class protocol along with other NAS protocols on top of its distributed OneFS file system.

COSBench is a popular benchmarking tool to measure the performance of Cloud Object Storage services and supports the S3 protocol. In the following blog, we will walk through how to set up COSBench to test the S3 performance of an PowerScale cluster.

Step 1：Choose v0.4.2.c4 version

I suggest choosing the v0.4.2 release candidate 4 instead of the latest v0.4.2 release, especially if you receive an error message like the following and your COSBench service cannot be started:

# cat driver-boot.log     
Listening on port 0.0.0.0/0.0.0.0:18089 ...
!SESSION 2020-06-03 10:12:59.683 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.7.0_261
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Command-line arguments:  -console 18089
!ENTRY org.eclipse.osgi 4 0 2020-06-03 10:13:00.367
!MESSAGE Bundle plugins/cosbench-castor not found.
!ENTRY org.eclipse.osgi 4 0 2020-06-03 10:13:00.368
!MESSAGE Bundle plugins/cosbench-log4j not found.
!ENTRY org.eclipse.osgi 4 0 2020-06-03 10:13:00.368
!MESSAGE Bundle plugins/cosbench-log@6:start not found.
!ENTRY org.eclipse.osgi 4 0 2020-06-03 10:13:00.369
!MESSAGE Bundle plugins/cosbench-config@6:start not found.

Step 2: Install Java

Both Java 1.7 and 1.8 work well with COSBench.

Step 3: Config ncat

Ncat is necessary for COSBench to work. Without it, you will receive the following error message:

[root]hopisdtmelabs14# bash ./start-driver.sh  
Launching osgi framwork ...
Successfully launched osgi framework!
Booting cosbench driver ...
which: no nc in (/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/tme/bin:/usr/local/tme/tme_portal/perf_web/bin)
No appropriate tool found to detect cosbench driver status.

Use the following commands to install Ncat (example here is CentOS 7) and config it for COSBench:

yum -y install wget
wget [https://nmap.org/dist/ncat-7.80-1.x86_64.rpm](https://nmap.org/dist/ncat-7.80-1.x86_64.rpm)
yum localinstall ncat-7.80-1.x86_64.rpm
cd /usr/bin
ln -s ncat nc

Step 4: Unzip the COSBench files

After you download the 0.4.2.c4.zip, you can unzip it to a directory:

unzip 0.4.2.c4.zip

Grant all the bash script permission to be executed:

chmod +x /tmp/cosbench/0.4.2.c4/*.sh

Step 5: Start drivers and controller

On drivers and controller, find the cosbench-start.sh. Locate the java launching line, then add the following two options:

-Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true
-Dcom.amazonaws.services.s3.disablePutObjectMD5Validation=true

The COSBench tool has two roles: controller and driver. You can use the following command to start the driver:

bash ./cosbench/start-driver.sh

Before we start the controller, we need to change the configuration to let the controller knows how many drivers it has and their addresses. This is done by filling in information in the controller's main configuration file. The configuration file is under ./conf, and the name of the file is controller.conf. Following is an example of the controller.conf:

[controller]
drivers = 4
log_level = INFO
log_file = log/system.log
archive_dir = archive
 
[driver1]
name = driver1
url = [http://10.245.109.115:18088/driver](http://10.245.109.115:18088/driver)
 
[driver2]
name = driver2
url = [http://10.245.109.116:18088/driver](http://10.245.109.116:18088/driver)
 
[driver3]
name = driver3
url = [http://10.245.109.117:18088/driver](http://10.245.109.117:18088/driver)
 
[driver4]
name = driver4
url = [http://10.245.109.118:18088/driver](http://10.245.109.118:18088/driver)

Run the start-controller.sh to start the controller role:

bash ./start-controller.sh

Step 6: Prepare PowerScale

First, you need to prepare your PowerScale cluster for the S3 test. Make sure to record the secret key of the newly created user, s3. Run the following commands to prepare PowerScale for the S3 performance test：

isi services s3 enable
isi s3 settings global modify --https-only=false
isi auth users create s3 --enabled=true
isi s3 keys create s3
mkdir -p -m 777 /ifs/s3/bkt1  
chmod 777 /ifs/s3
isi s3 buckets create --owner=s3 --name=bkt1  --path=/ifs/s3/bkt1

Compose the workload XML file, and use it to specify the details of the test you want to run. Here is an example:

<?xml version="1.0" encoding="UTF-8"?>
<workload name="S3-F600-Test1" description="Isilon F600 with original configuration">
        <storage type="s3" config="accesskey=1_s3_accid;secretkey=wEUqWNWkQGmgMos70NInqW26WpGf;endpoint=http://f600-2:9020/bkt1;path_style_access=true"/>
        <workflow>
               <workstage name="init-for-write-1k">
                       <work type="init" workers="1" config="cprefix=write-bucket-1k; containers=r(1,6)"/>
               </workstage>
               <workstage name="init-for-read-1k">
                       <work type="init" workers="1" config="cprefix=read-bucket-1k; containers=r(1,6)"/>
               </workstage>
               <workstage name="prepare-1k">
                       <work type="prepare" workers="1" config="cprefix=read-bucket-1k;containers=r(1,6);oprefix=1kb_;objects=r(1,1000);sizes=c(1)KB"/>
               </workstage>
               <workstage name="write-1kb">
                       <work name="main" type="normal" interval="5" division="container" chunked="false" rampup="0" rampdown="0" workers="6" totalOps="6000">
                               <operation type="write" config="cprefix=write-bucket-1k; containers=r(1,6); oprefix=1kb_; objects=r(1,1000); sizes=c(1)KB"/>
                       </work>
               </workstage>
               <workstage name="read-1kb">
                       <work name="main" type="normal" interval="5" division="container" chunked="false" rampup="0" rampdown="0" workers="6" totalOps="6000">
                               <operation type="read" config="cprefix=read-bucket-1k; containers=r(1,6); oprefix=1kb_; objects=r(1,1000)"/>
                       </work>
               </workstage>
        </workflow>
</workload>

Step 7: Run the test

You can directly submit the XML in the COSBench WebUI, or you can use the following command line in the controller console to start the test:

bash ./cli.sh submit ./conf/my-s3-test.xml

You will see the test successfully finished, as shown in the following figure.

Figure 1. Completion screen after testing

Have fun testing!

Author: Yunlong Zhang

Dell Technologies has developed a range of PowerScale platforms, including all flash models, hybrid models, and archive models, all of which exhibit exceptional design. The synergy between the disk system and the compute system is highly effective, showcasing a well-matched integration.

In the cloud environment, customers have the flexibility to control the number of CPU cores and memory sizes by selecting different instance types. APEX File Storage for AWS uses EBS volumes as its node disks. Customers can also select a different number of EBS volumes in each node, and for gp3 volumes, customers are able to customize the performance of each volume by specifying the throughput or IOPS capability.

With this level of flexibility, how shall we configure the disk system to make the most out of the entire OneFS system? Typically, in an on-prem appliance, the more disks a PowerScale node contains, the better performance the disk system can provide thanks to a greater number of devices contributing to the delivery of throughput or IOPS.

In a OneFS cloud environment, does it hold true that more EBS volumes indicates better performance? In short, it depends. When the aggregated EBS volume performance is smaller than the instance EBS bandwidth limit, test results show that more EBS volumes can improve performance. When aggregated EBS volume performance is larger than EBS bandwidth limit, adding more EBS volumes will not improve performance.

What is the best practice of setting the number of EBS volumes of each node?

1. Make the aggregated EBS volume bandwidth limit match the instance type EBS bandwidth limit. 

For example, we want to use m5dn.16xlarge as the instance type of our OneFS cloud system. According to AWS, the EBS Bandwidth of m5dn.16xlarge is 13,600 Mbps, which is 1700 MB/sec. If we choose to use 10 EBS volumes in each node, then we should config each gp3 EBS volume to be capable of delivering 170 MB/sec throughput. This will make the aggregated EBS volume throughput equal to the m5dn.16xlarge EBS bandwidth limit.

Note that each gp3 EBS volume has 125MB/sec free throughput and 3,000 IOPS for free. As a cost-saving measure, we can config each node to have 12 EBS volumes to better leverage free EBS volume throughput.

 For example, considering an m5dn.16xlarge instance type with 12 TB raw capacity per node, the disk cost of 10 volumes and 12 volumes are as follows:

1. For 10 drives, each EBS volume should support 170 MB/sec throughput, and each node EBS storage cost is 1001.2 USD a month.
2. For 12 drives, each EBS volume should support 142 MB/sec throughput, and each node EBS storage cost is 991.20 USD a month.

Using 12 EBS volumes can save $10 per node per month.

2. Do not set up more than 12 EBS volumes in each node.

Although APEX File Storage for AWS also supports 15, 18, and 20 gp3 volumes in each node, we do not recommend configuring more than 12 EBS volumes in each node for OneFS 9.7. This is best practice for keeping software journal space for each disk from being too small and is beneficial for write performance.  

Author: Yunlong Zhang

In the first release of APEX File Storage for AWS in May 2023, users gained the capability to execute file workloads in the AWS cloud, thus harnessing the power of the PowerScale OneFS scale-out NAS storage solution. However, the initial implementation required the manual provisioning of all necessary AWS resources to provision the OneFS cluster—a less than optimal experience for embarking on the APEX File Storage journey in AWS.

With the subsequent release of APEX File Storage for AWS in December 2023, we are pleased to introduce a new, user-friendly open-source Terraform module. This module is designed to enhance and simplify the deployment process, alleviating the need for manual resource provisioning. In this blog post, we will delve into the details of leveraging this Terraform module, providing you with a comprehensive guide to expedite your APEX File Storage deployment on AWS.

Overview of Terraform onefs module

Terraform onefs module is an open-source module for the auto-deployment of AWS resources for a OneFS cluster. It is released and licensed under the MPL-2.0 license. You can find more details on the onefs module from the Terraform Registry. The onefs module provides the following features to help you deploy APEX File Storage for AWS OneFS clusters in AWS:

• Provision necessary AWS resources for a single OneFS cluster, including EC2 instances, EBS volumes, placement group, and network interfaces.
• Expand cluster size by provisioning additional AWS resources, including EC2 instances, EBS volumes, and network interfaces.

Getting Started

To use the Terraform onefs module, you need a machine that has Terraform installed and can connect to your AWS account. After you have fulfilled the prerequisites in documentation, you can start to deploy AWS resources for a OneFS cluster.  

This blog provides instructions for deploying the required AWS infrastructure resources for APEX File Storage for AWS with Terraform.This includes: EC2 instances, spread strategy placement group, network interfaces, and EBS volumes.

1. Get the latest version of the onefs module from the Terraform Registry.

2. Prepare a main.tf file that uses the onefs module version collected in Step 1. The onefs module requires a set of input variables. The following is an example file named main.tf for creating a 4-nodes OneFS cluster.

module "onefs" {

   source  = "dell/onefs/aws"

   version = "1.0.0"

   region = "us-east-1"

   availability_zone = "us-east-1a"

   iam_instance_profile = "onefs-runtime-instance-profile"

   name = "vonefs-cfv"

   id = "vonefs-cfv"

   nodes = 4

   instance_type = "m5dn.12xlarge"

   data_disk_type = "gp3"

   data_disk_size = 1024

   data_disks_per_node = 6

   internal_subnet_id = "subnet-0c0106598b95ee7b6"

   external_subnet_id = "subnet-0837801239d54e245"

   contiguous_ips= true

   first_external_node_hostnum = 5

   internal_sg_id = "sg-0ee87249a52397219"

   security_group_external_id = "sg-0635f298c9cb764da"

   image_id = "ami-0f1a267119a34361c"

   credentials_hashed = true

   hashed_root_passphrase = "$5$9874f5d2c724b8ca$IFZZ5e9yfUVqNKVL82s.iFLIktr4WLavFhUVa8A"

   hashed_admin_passphrase = "$5$9874f5d2c724b8ca$IFZZ5e9yfUVqNKVL82s.iFLIktr4WLavFhUVa8A"

   dns_servers = ["169.254.169.253"]

   timezone = "Greenwich Mean Time"

}

output "onefs-outputs" {

   value = module.onefs

   sensitive = true

}

3. Change your current working directory to the main.tf directory.

4. Initialize the module’s root directory by installing the required providers and modules for the deployment. In the following example, the onefs module is downloaded automatically from the Terraform Registry.

# terraform init

Initializing the backend...

Initializing modules...

Downloading registry.terraform.io/dell/onefs/aws 1.0.0 for onefs...

- onefs in .terraform\modules\onefs

- onefs.onefsbase in .terraform\modules\onefs\modules\base

- onefs.onefsbase.machineid in .terraform\modules\onefs\modules\machineid

Initializing provider plugins...

- Finding latest version of hashicorp/aws...

- Installing hashicorp/aws v5.30.0...

- Installed hashicorp/aws v5.30.0 (signed by HashiCorp)

5. Verify the configuration files in the onefs directory.

# terraform validate

6. Apply the configurations by running the following command.

# terraform apply

7. Enter “yes” after you have previewed and confirmed the changes.

Do you want to perform these actions?

   Terraform will perform the actions described above.

   Only 'yes' will be accepted to approve.

   Enter a value: yes

8. Wait for the AWS resources to be provisioned. The output displays all the cluster information. If the deployment fails, re-run the terraform apply command to deploy.

Apply complete! Resources: 13 added, 0 changed, 0 destroyed.

Outputs:

onefs-outputs = <sensitive>

9. Get the cluster details information by running the following command.

# terraform output --json

The following example output is truncated.

additional_nodes = 3

cluster_id = "vonefs-cfv"

control_ip_address = "10.0.32.5"

external_ip_addresses = [

   "10.0.32.5",

   "10.0.32.6",

   "10.0.32.7",

   "10.0.32.8",

]

gateway_hostnum = 1

instance_id = [

   "i-0eead1ee1dd67da6e",

   "i-054efe96f6e605009",

   "i-06e0b1ce06bad42a1",

   "i-0e463c742974641d7",

]

internal_ip_addresses = [

   "10.0.16.5",

   "10.0.16.6",

   "10.0.16.7",

   "10.0.16.8",

]

internal_network_high_ip = "10.0.16.8"

internal_network_low_ip = "10.0.16.5"

mgmt_ip_addresses = []

node_configs = {

   "0" = {

     "external_interface_id" = "eni-09ddea1fd79f0d0ab"

     "external_ips" = [

       "10.0.32.5",

     ]

     "internal_interface_id" = "eni-0caeee71581a8c429"

     "internal_ips" = [

       "10.0.16.5",

     ]

     "mgmt_interface_id" = null

     "mgmt_ips" = null /* tuple */

     "serial_number" = "SV200-930073-0000"

   }

   "1" = {

     "external_interface_id" = "eni-00869c96a27c20c93"

     "external_ips" = [

       "10.0.32.6",

     ]

     "internal_interface_id" = "eni-0471bbba5a7f6596d"

     "internal_ips" = [

       "10.0.16.6",

     ]

     "mgmt_interface_id" = null

     "mgmt_ips" = null /* tuple */

     "serial_number" = "SV200-930073-0001"

   }

   "2" = {

     "external_interface_id" = "eni-0dac5052668bd3a4f"

     "external_ips" = [

       "10.0.32.7",

     ]

     "internal_interface_id" = "eni-09d35ffa61b3dcd60"

     "internal_ips" = [

       "10.0.16.7",

     ]

     "mgmt_interface_id" = null

     "mgmt_ips" = null /* tuple */

     "serial_number" = "SV200-930073-0002"

   }

   "3" = {

     "external_interface_id" = "eni-028d211ef2d5b577c"

     "external_ips" = [

       "10.0.32.8",

     ]

     "internal_interface_id" = "eni-02a99febea713f2d1"

     "internal_ips" = [

       "10.0.16.8",

     ]

     "mgmt_interface_id" = null

     "mgmt_ips" = null /* tuple */

     "serial_number" = "SV200-930073-0003"

   }

}

region = "us-east-1"

10. Write down the following output variables for setting up a cluster described in documentation.

• control_ip_address: The external IP address of the cluster’s first node
• external_ip_addresses: The external IP addresses of all provisioned cluster nodes
• internal_ip_addresses: The internal IP addresses of all provisioned cluster nodes
• internal_network_high_ip: The highest internal IP address assigned
• internal_network_low_ip: The lowest internal IP address assigned
• instance_id: The EC2 instance IDs of the cluster nodes

11. All AWS resources are now provisioned. After the cluster’s first node starts, it will form a single node cluster. You can use the cluster’s first node to add additional nodes to the cluster described in documentation. Below are the provisioned AWS EC2 instances with Terraform onefs module.

Available input variables

The Terraform onefs module provides a set of input variables for you to specify your own settings, including AWS resources and OneFS cluster, for example: AWS network resources, cluster name and password. See the table below for details used in the main.tf file. 

Learn More

In this article, we have shown how to use Terraform onefs module. You can refer to the documentation below for more details about APEX File Storage for AWS:

• APEX File Storage for AWS
• Terraform onefs Module 
• Technical white paper for AI use case:  APEX File Storage for AWS with Amazon SageMaker   
• Technical White Paper for M&E use case: APEX File Storage for AWS for Video Edit in AWS   
• APEX File Storage for AWS Manual Deployment Guide
• APEX File Storage for AWS Deployment Guide with Terraform
• APEX File Storage for AWS Interactive Demo

Author: Lieven Lin

NDMP (Network Data Management Protocol) specifies a common architecture and data format for backups and restores of NAS (Network Attached Storage), allowing heterogeneous network file servers to directly communicate to tape devices for backup and restore operations. NDMP addresses the problems caused by the integrations of different backup software or DMA (Data Management Applications), file servers, and tape devices.  

The NDMP architecture is a client/server model with the following characteristics:

• The NDMP host is a file server that is being protected with an NDMP backup solution.
• The NDMP server is a virtual state machine on the NDMP host that is controlled using NDMP.
• The backup software is considered as a client to the NDMP server.

OneFS supports the following two types of NDMP backups:

• NDMP two-way backup
• NDMP three-way backup

In both backup models, OneFS takes a snapshot of the backup directory to ensure consistency of data. The backup operates on the snapshot instead of the source directory, which allows users to continue read/write activities as normal. OneFS makes entries in the file history that are transferred from the PowerScale cluster to the backup server during the backup.

NDMP two-way backup

The NDMP two-way backup is also known as the local or direct NDMP backup, which is considered the most efficient model and usually provides the best performance. The backup moves the backup data directly from the PowerScale cluster to the tape devices without moving to the backup server over the network.

In this model, OneFS must detect the tape devices before you back up data to them. The PowerScale cluster provides the option for NDMP two-way backups as shown in the following figure. You can connect the PowerScale cluster to a Backup Accelerator node and connect tape devices to that node. The Backup Accelerator node is synonymous with a Fibre Attached Storage node without adding primary storage and offloads NDMP workloads from the primary storage nodes. You can directly connect tape devices to the Fibre Channel ports on the PowerScale cluster or Backup Accelerator node using Fibre Channel. Alternatively, you can connect Fibre Channel switches to the Fibre Channel ports that connect tape devices to the PowerScale cluster or Backup Accelerator node.

 Figure 1. NDMP two-way backup with B100 backup accelerator connected to the PowerScale cluster

The following table shows details of the NDMP two-way backup supported by PowerScale:  

Note: The B100 backup accelerator requires OneFS 9.3.0.0 or later.

 NDMP three-way backup

The NDMP three-way backup, also known as the remote NDMP backup, is shown in the following figure.

Figure 2. NDMP three-way backup

In this backup mode, the tape devices are connected to the backup media server. OneFS does not detect tape devices on the PowerScale cluster, and Fibre Channel ports are not required on the PowerScale cluster. The NDMP service runs on the NDMP server or the PowerScale cluster. The NDMP tape service runs on the backup media server. A DMA on the backup server instructs the PowerScale cluster to start backing up data from the PowerScale cluster to the backup media server over the network. The backup media server moves the backup data to tape devices. Both servers are connected to each other across the network boundary. Sometimes, the backup server and backup media server reside on the same physical machine.

For some specific DMA, DMA can write NDMP data to non-NDMP devices. For example, Dell NetWorker software writes NDMP data to non-NDMP devices, including tape, virtual tape, Advanced File Type Device (AFTD), and Dell PowerProtect DD series appliances. For more information on Data Protection with Dell NetWorker using NDMP, refer to this guide: Dell PowerScale: Data Protection with Dell NetWorker using NDMP.

Author: Jason He, Principal Engineering Technologist

We are thrilled to announce the latest version of APEX File Storage for AWS! This release brings a multitude of enhancements to elevate your AWS file storage experience, including expanded AWS regions with the support for additional EC2 instance types, a Terraform module for streamlined deployment, larger raw capacity, and additional OneFS features support.

APEX File Storage delivers Dell’s leading enterprise-class high-performance scale-out file storage as a software-defined customer-managed offer in the public cloud. Based on PowerScale OneFS, APEX File Storage for AWS brings enterprise file capabilities and performance and delivers operational consistency across multicloud environments, simplifying hybrid cloud environments by facilitating seamless data mobility between on-premises and the cloud with native replication and making it the perfect option to run AI workloads. APEX File Storage can enhance customers’ development and innovation initiatives by combining proven data services such as multi-protocol access, security features, and a proven scale-out architecture with the flexibility of public cloud infrastructure and services. APEX File Storage enables organizations to run the software they trust directly in the public cloud without retraining their staff or refactoring their storage architecture.

What's New?

1. Additional EC2 instance types support

We've expanded compatibility by adding support for a wider range of EC2 instance types. This means you have more flexibility in choosing the instance type that best suits your performance and resource requirements. We now support the following EC2 instance types:

• EC2 m5dn instances: m5dn.8xlarge, m5dn.12xlarge, m5dn.16xlarge, m5dn.24xlarge
• EC2 m6idn instances: m6idn.8xlarge, m6idn.12xlarge, m6idn.16xlarge, m6idn.24xlarge
• EC2 m5d instances: m5d.24xlarge
• EC2 i3en instances: i3en.12xlarge

Please note that it is required to run PoC if you intend to use m5d.24xlarge or i3en.12xlarge EC2 instance types. Please contact your Dell account team for the details.

2. Extended AWS regions support

APEX File Storage is now available in more AWS regions than ever before. A total of 28 regions are available for you. We understand that our users operate globally, and this expansion ensures that you can leverage APEX File Storage wherever your AWS resources are located. The following table lists all available regions for different EC2 instance types:

3. Terraform module: auto-deployment made effortless

Simplify your deployment process with our new Terraform module, which automates the AWS resource deployment process to ensure a smooth and error-free experience.

Once you fulfill the deployment prerequisites, you can deploy a cluster with a single Terraform command. For more details, refer to documentation: APEX File Storage for AWS Deployment Guide with Terraform. Stay tuned for a blog with additional details coming soon. 

4. Larger raw capacity: more room for your data

Your data is growing, and so should your storage capacity. APEX File Storage for AWS can now support up to 1.6PiB raw capacity, enabling workloads that produce a vast amount of data such as AI and ensuring that you have ample space to store, manage, and scale your data effortlessly.

5. Additional OneFS features support

The OneFS features not supported in the first release of APEX File Storage for AWS are now supported, including:

• Enhanced Protocols: With HDFS protocol support, you can seamlessly integrate HDFS into your workflows, enhancing your data processing capabilities in AWS. Enjoy expanded connectivity with support for HTTP and FTP protocols, providing more flexibility in accessing and managing your files.
• Quality of Service – SmartQoS: Ensure a consistent and reliable user experience with SmartQoS, which enables you to prioritize workloads and applications based on performance requirements.
• Immutable Data Protection - SmartLock: Enhance data protection by leveraging SmartLock to create Write Once Read Many (WORM) files, providing an added layer of security against accidental or intentional data alteration.
• Large File Support: Address the needs of large-scale data processing with improved support for large files, facilitating efficient storage and retrieval. A single file size can be up to 16TiB now.

Learn More

For deployment instructions and detailed information on these exciting new features, refer to our documentation:

• APEX File Storage for AWS
• terraform-aws-onefs Terraform Module
• Technical white paper for AI use case:  APEX File Storage for AWS with Amazon SageMaker    
• Technical White Paper for M&E use case: APEX File Storage for AWS for Video Edit in AWS   
• APEX File Storage for AWS Manual Deployment Guide
• APEX File Storage for AWS Deployment Guide with Terraform
• APEX File Storage for AWS Interactive Demo

Author: Lieven Lin

REST APIs have been introduced in IIQ 5.0.0, providing the equivalent of the CLI command in the previous IIQ version. CLI will not be available in IIQ 5.0.0. In order to understand how REST APIs work in IIQ, we will cover:

• REST API Authentication
• Creating a REST API Session
• Getting a REST API Session
• Managing PowerScale Clusters using REST API
• Exporting a Performance Report
• Deleting a REST API Session

Let’s get started!

REST API Authentication

IIQ 5.0.0 leverages JSON Web Tokens (JWT) along with x-csrf-token for session-based authentication. Following are some of the benefits of using JWTs:

• JWTs contains the user’s details
• JWTs incorporate digital signatures to ensure their integrity and protect against unauthorized modifications by potential attackers
• JWTs offer efficiency and rapid verification processes

Creating a REST API Session

A POST request to /insightiq/rest/security-iam/v1/auth/login/ will create a session with JWT cookie and x-csrf-token. A status code of 201 (Created) is returned upon successful user authentication. If the authentication process fails, the API responds with a status code of 401 (Unauthorized).

The following is an example of getting the JWT token with the POST method.

The POST request is:

curl -vk -X POST https://172.16.202.71:8000/insightiq/rest/security-iam/v1/auth/login -d '{"username": "administrator", "password": "a"}'  -H 'accept: application/json'  -H 'Content-Type: application/json'

The POST response is:

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 172.16.202.71:8000...
* Connected to 172.16.202.71 (172.16.202.71) port 8000 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: O=Test; CN=cmo.ingress.dell
*  start date: Dec  4 05:59:14 2023 GMT
*  expire date: Dec  4 07:59:44 2023 GMT
*  issuer: C=US; ST=TX; L=Round Rock; O=DELL EMC; OU=Storage; CN=Platform Root CA; emailAddress=a@dell.com
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/2
* h2h3 [:method: POST]
* h2h3 [:path: /insightiq/rest/security-iam/v1/auth/login]
* h2h3 [:scheme: https]
* h2h3 [:authority: 172.16.202.71:8000]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: application/json]
* h2h3 [content-type: application/json]
* h2h3 [content-length: 46]
* Using Stream ID: 1 (easy handle 0x5618836b5eb0)
> POST /insightiq/rest/security-iam/v1/auth/login HTTP/2
> Host: 172.16.202.71:8000
> user-agent: curl/8.0.1
> accept: application/json
> content-type: application/json
> content-length: 46
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
< HTTP/2 201
< server: istio-envoy
< date: Mon, 04 Dec 2023 07:19:19 GMT
< content-type: application/json
< content-length: 54
< set-cookie: insightiq_auth=eyJ0eXAiOiJKV1QiLCJhbGciOiJQUzUxMiJ9.eyJjc3JmIjoiN3Z0eG5sMWRxbHIzaGtubGp3MjdwYXl3eW54bzQzdGs0Zmx4IiwiZXhwIjoxNzAxNzg5MTM5LCJpYXQiOjE3MDE3NDU5MzksImlzcyI6IkRlbGwgVGVjaG5vbG9naWVzIiwicm9sZSI6ImFkbWluIiwic2Vzc2lvbiI6InJ6ZnA3ZTRpMXdzd2xuYjBuNGo3YmQwNmF5dWs3emNkeXp1ZSIsInN1YiI6ImFkbWluaXN0cmF0b3IifQ.yKyfXbezscqn6UPa9fXxxjh71MCgeRAXPZhXkG-v92siwXAEP40ASb5bQUFHnAmWwwtlB4Jt8lX9kY8LmRkqi1V7B3v0LgxUp68heAc0HZAh6XO92ac9AfZ9dAuE9H3U4RNELm4vVx8mGrGmuzQymWUG5yRCNk03SpeW8esHnTPRVoGGsE4Cf6ta3BrUXBfic-D_TL01YgyY3Dy_T8Z1oqhkD508GPEYnEeNMU1QtZAwkmj6MJHtGmp69T0ljtQdIW2oi5xYdPs-ZHGSFRGG4j2o8xAEFV8A4igzP-5XOkE9NCcx2mkj67OdvVgNBxCcY-X7cnYyfLgagkanyQSgdA; Secure; HttpOnly; Path=/
< set-cookie: csrf_token=7vtxnl1dqlr3hknljw27paywynxo43tk4flx; Secure; HttpOnly; Path=/
< x-csrf-token: 7vtxnl1dqlr3hknljw27paywynxo43tk4flx
< x-envoy-upstream-service-time: 3179
< content-security-policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' data:; style-src 'unsafe-inline' 'self';
< x-frame-options: sameorigin
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< referrer-policy: strict-origin-when-cross-origin
< 
{"timeout_absolute":43200,"username":"administrator"}
* Connection #0 to host 172.16.202.71 left intact

The JWT cookie and x-csrf-token have been created and highlighted in the POST response section. The timeout for the session is 43,200 seconds (12 hours). You can save them for future uses:

export TOK="insightiq_auth=eyJ0eXAiOiJKV1QiLCJhbGciOiJQUzUxMiJ9.eyJjc3JmIjoiN3Z0eG5sMWRxbHIzaGtubGp3MjdwYXl3eW54bzQzdGs0Zmx4IiwiZXhwIjoxNzAxNzg5MTM5LCJpYXQiOjE3MDE3NDU5MzksImlzcyI6IkRlbGwgVGVjaG5vbG9naWVzIiwicm9sZSI6ImFkbWluIiwic2Vzc2lvbiI6InJ6ZnA3ZTRpMXdzd2xuYjBuNGo3YmQwNmF5dWs3emNkeXp1ZSIsInN1YiI6ImFkbWluaXN0cmF0b3IifQ.yKyfXbezscqn6UPa9fXxxjh71MCgeRAXPZhXkG-v92siwXAEP40ASb5bQUFHnAmWwwtlB4Jt8lX9kY8LmRkqi1V7B3v0LgxUp68heAc0HZAh6XO92ac9AfZ9dAuE9H3U4RNELm4vVx8mGrGmuzQymWUG5yRCNk03SpeW8esHnTPRVoGGsE4Cf6ta3BrUXBfic-D_TL01YgyY3Dy_T8Z1oqhkD508GPEYnEeNMU1QtZAwkmj6MJHtGmp69T0ljtQdIW2oi5xYdPs-ZHGSFRGG4j2o8xAEFV8A4igzP-5XOkE9NCcx2mkj67OdvVgNBxCcY-X7cnYyfLgagkanyQSgdA"

Getting a REST API Session

Use the GET method against /insightiq/rest/security-iam/v1/auth/session/ to get the session information. In the request header, include the cookie and the x-csrf-token field for authentication.

curl -k -v -X GET https://172.16.202.71:8000/insightiq/rest/security-iam/v1/auth/session --cookie $TOK -H 'accept: application/json'  -H 'Content-Type: application/json' -H 'x-csrf-token: 7vtxnl1dqlr3hknljw27paywynxo43tk4flx'

The response is:

Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 172.16.202.71:8000...
* Connected to 172.16.202.71 (172.16.202.71) port 8000 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: O=Test; CN=cmo.ingress.dell
*  start date: Dec  5 03:16:34 2023 GMT
*  expire date: Dec  5 05:17:04 2023 GMT
*  issuer: C=US; ST=TX; L=Round Rock; O=DELL EMC; OU=Storage; CN=Platform Root CA; emailAddress=a@dell.com
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /insightiq/rest/security-iam/v1/auth/session]
* h2h3 [:scheme: https]
* h2h3 [:authority: 172.16.202.71:8000]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [cookie: insightiq_auth=eyJ0eXAiOiJKV1QiLCJhbGciOiJQUzUxMiJ9.eyJjc3JmIjoiN3Z0eG5sMWRxbHIzaGtubGp3MjdwYXl3eW54bzQzdGs0Zmx4IiwiZXhwIjoxNzAxNzg5MTM5LCJpYXQiOjE3MDE3NDU5MzksImlzcyI6IkRlbGwgVGVjaG5vbG9naWVzIiwicm9sZSI6ImFkbWluIiwic2Vzc2lvbiI6InJ6ZnA3ZTRpMXdzd2xuYjBuNGo3YmQwNmF5dWs3emNkeXp1ZSIsInN1YiI6ImFkbWluaXN0cmF0b3IifQ.yKyfXbezscqn6UPa9fXxxjh71MCgeRAXPZhXkG-v92siwXAEP40ASb5bQUFHnAmWwwtlB4Jt8lX9kY8LmRkqi1V7B3v0LgxUp68heAc0HZAh6XO92ac9AfZ9dAuE9H3U4RNELm4vVx8mGrGmuzQymWUG5yRCNk03SpeW8esHnTPRVoGGsE4Cf6ta3BrUXBfic-D_TL01YgyY3Dy_T8Z1oqhkD508GPEYnEeNMU1QtZAwkmj6MJHtGmp69T0ljtQdIW2oi5xYdPs-ZHGSFRGG4j2o8xAEFV8A4igzP-5XOkE9NCcx2mkj67OdvVgNBxCcY-X7cnYyfLgagkanyQSgdA]
* h2h3 [accept: application/json]
* h2h3 [content-type: application/json]
* h2h3 [x-csrf-token: 7vtxnl1dqlr3hknljw27paywynxo43tk4flx]
* Using Stream ID: 1 (easy handle 0x561f902392c0)
> GET /insightiq/rest/security-iam/v1/auth/session HTTP/2
> Host: 172.16.202.71:8000
> user-agent: curl/8.0.1
> cookie: insightiq_auth=eyJ0eXAiOiJKV1QiLCJhbGciOiJQUzUxMiJ9.eyJjc3JmIjoiN3Z0eG5sMWRxbHIzaGtubGp3MjdwYXl3eW54bzQzdGs0Zmx4IiwiZXhwIjoxNzAxNzg5MTM5LCJpYXQiOjE3MDE3NDU5MzksImlzcyI6IkRlbGwgVGVjaG5vbG9naWVzIiwicm9sZSI6ImFkbWluIiwic2Vzc2lvbiI6InJ6ZnA3ZTRpMXdzd2xuYjBuNGo3YmQwNmF5dWs3emNkeXp1ZSIsInN1YiI6ImFkbWluaXN0cmF0b3IifQ.yKyfXbezscqn6UPa9fXxxjh71MCgeRAXPZhXkG-v92siwXAEP40ASb5bQUFHnAmWwwtlB4Jt8lX9kY8LmRkqi1V7B3v0LgxUp68heAc0HZAh6XO92ac9AfZ9dAuE9H3U4RNELm4vVx8mGrGmuzQymWUG5yRCNk03SpeW8esHnTPRVoGGsE4Cf6ta3BrUXBfic-D_TL01YgyY3Dy_T8Z1oqhkD508GPEYnEeNMU1QtZAwkmj6MJHtGmp69T0ljtQdIW2oi5xYdPs-ZHGSFRGG4j2o8xAEFV8A4igzP-5XOkE9NCcx2mkj67OdvVgNBxCcY-X7cnYyfLgagkanyQSgdA
> accept: application/json
> content-type: application/json
> x-csrf-token: 7vtxnl1dqlr3hknljw27paywynxo43tk4flx
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
< server: istio-envoy
< date: Tue, 05 Dec 2023 03:19:41 GMT
< content-type: application/json
< content-length: 57
< x-envoy-upstream-service-time: 5
< content-security-policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' data:; style-src 'unsafe-inline' 'self';
< x-frame-options: sameorigin
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< referrer-policy: strict-origin-when-cross-origin
< 
{"username": "administrator", "timeout_absolute": 42758}
* Connection #0 to host 172.16.202.71 left intact

The response body will return the username of the session and its remaining timeout value in seconds.

Managing PowerScale Clusters using REST API

You can also use the IIQ REST API to manage your PowerScale cluster. Like what we’ve seen so far, all requests must include the parameter cookie and x-csrf-token for authentication. When adding clusters into IIQ, you will also need to provide the cluster IP with username and password. For details, refer to the following table:

Table 1. Using REST API to manage PowerScale Clusters

POST /insightiq/rest/clustermanager/v1/clusters

curl -k -v -X 'POST' \
'https://<EXTERNAL_IP>:8000/insightiq/rest/clustermanager/v1/clusters' \
  --cookie <COOKIE> \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'x-csrf-token: <X-CSRF-TOKEN>'
  -d '{
    "host": "<HOST>",
    "username": "<USERNAME>",
    "password": "<PASSWORD>"
}'

DELETE /insightiq/rest/clustermanager/v1/clusters/<GUID>

curl -k -v -X 'DELETE' \
'https://<EXTERNAL_IP>:8000/insightiq/rest/clustermanager/v1/clusters/<GUID>' \
--cookie <COOKIE> \
-H 'accept: application/json' \
-H 'x-csrf-token: <X-CSRF-TOKEN>'

Exporting a Performance Report

To export a performance report from IIQ 5.0.0, you can use the GET method against /iiq/reporting/api/v1/timeseries/download_data with the following query parameters:

• cluster – PowerScale cluster id 
• start_time – UNIX epoch timestamp of the beginning of the data range. Defaults to most recent saved report data and time.
• end_time – UNIX epoch timestamp of the end of the data range. Defaults to most recent saved report data and time.
• key – the performance key. To get a list of all the supported keys, use the GET method against http://<lP>:8000/iiq/reporting/api/v1/reports/data-element-types

Note: To get the performance key, use values from the response list for data-element-types in definition.timeseries_keys from data elements where report_group is equal to performance and definition.layout is equal to chart. See the following screenshot for an example. In this example, the performance key is ext_net.

 Figure 1. Getting the performance key where the key is ext_net

The following is an example of using the IIQ REST API to export the cluster performance of external network throughput to a CSV file:

curl -vk -X GET "https://10.246.159.113:8000/iiq/reporting/api/v1/timeseries/download_data?cluster=0007433384d03e80b4582103b56e1cac33a2&start_time=1694143511&end_time=1694366711&key=ext_net" -H 'x-csrf-token: vny27rem4l6ww29hvkhuaka0ix7x172wufbv' --cookie $TOK >>perf.csv

Deleting a REST API Session

To remove an IIQ REST API session, use the following API:

curl -k -v -X GET https://<EXTERNAL_IP>:8000/insightiq/rest/security-iam/v1/auth/logout --cookie <COOKIE> -H 'accept: application/json'  -H 'Content-Type: application/json' -H 'x-csrf-token: <X-CSRF-TOKEN>'

Conclusion

The IIQ REST API is a powerful tool. Please refer to the Dell Technologies InsightIQ 5.0.0 User Guide for more information. For any questions, feel free to reach out to me at Vincent.shen@dell.com. 

Author: Vincent Shen

Alert is a new feature introduced with the release of IIQ 5.0.0. It provides the capability and flexibility to configure alerts based on the KPI threshold.

This blog will walk you through the following aspects of this feature:

1. Introduction to Alert
2. How to configure alerts using Alert

Let’s get started:

Introduction

IIQ 5.0.0 can send email alerts based on your defined KPI and threshold. The supported KPIs are listed in the following table:

Each KPI requires a threshold and a severity level, together forming an alert rule. You can customize the alert rules to align with specific business use cases.

Here is an example of an alert rule:

If CPU usage (KPI) is greater than or equal to 96% (threshold), a critical alert (severity) will be triggered.

The supported severities are:

1. Emergency
2. Critical
3. Warning
4. Information

You can combine multiple alert rules into a single alert policy for easy management purposes.

If you take a look at the chart above, you will find a new concept called Notification Rule. This is used to define the recipients' Email address and from what severity they will receive an Email:

An example of a notification rule is like this: for user A (user_a@lled.com) and user B (user_b@lled.com), they both will receive Email alerts from all severity.

If you combine the above two examples and put them into the view of alert policy, you will get:

At this point, you should understand the  big picture of the alert feature in IIQ 5.0.0. In my next post, I will walk you through the details of how to configure it.

My previous post introduced one of the key features in IIQ 5.0.0 – Alert and explained how it works. In this blog, we will go into the details of how to configure it.  

How to configure an alert in IIQ 5.0.0

Configure SMTP server in IIQ

Follow the steps below to add the SMTP server in IIQ:

1. Access Configure SMTP under Settings from the left side menu.
2. Enter the SMTP Server IP or FQDN. Username and Password are optional.
3. Click the Save button.

You can send a test email to verify the settings.

1. SMTP configuration

Note: If you keep the SMTP Port number blank, the default will be 25 or 587 for TLS.

Manage Alerts

Create Alert Rules

To create alert rules, follow these steps:

1. Navigate to Manage Alerts under Alerts from the left side menu.
2. Click the Alert Rules.
3. Click the Create Alert Rule button and a pop-up window will appear as shown below:

1. Create Alert Rule
2. Specify the KPI, Severity, and Threshold for it. Click the Save button.
3. (Optional) You can create multiple Alert Rules.

Create a Notification Rule

A notification rule specifies the recipient(s) of SMTP alerts and its associated alert severity. To create a notification rule, follow these steps:

1. Navigate to Manage Alerts located below  Alerts from the left side menu.
2. Click the Notification Rules.
3. Click the button Create Notification Rule and it will pop up a window as shown below.

1. Create Notification Rule
2. Input the Recipient Email ID(s) and choose the severity from the dropdown list of Receive Emails for.
3. Click the Save button.

Create an alert policy

To create an alert policy, follow these steps:

1. Navigate to Manage Alerts located under Alerts from the left side menu.
2. Click the Alert Policies.
3. Click the Create Policy button.
4. Input the Name and Description in the Policy Details window and click the Next button.
5. In the Alert Rules subpage, you can choose either Existing Alert Rules or Create Alert Rule by clicking the corresponding button. After you create the alert rules, click the Next Button.

1. Add Alert Rules
2. On the Cluster subpage, choose the cluster to which you want to apply the alert settings, and click the Next button.

1. Choose clusters
2. On the Notification Rules subpage, you can choose either Existing Notification Rules or Create Notification Rule by clicking the corresponding button. After you choose the rule, click the Next button.

1. Specify Notification Rules
2. Click the Save button in the final Review subpage.
3. The following screenshot is a sample alert email:

1. Sample email alert

View Alerts

All the alerts can be accessed in Alerts > View Alerts from the left side menu.

1. View Alert

On this page you can:

1. Filter alerts by selecting the Duration.
2. Show alerts by choosing specific Clusters.
3. Categorize alerts by different severity levels.
4. Sort alerts by the Date & Time.

Hope you enjoy the reading. If you have any questions for suggestion on this feature, please feel free to reach out to me. (Vincent.shen@dell.com)

Overview

In the complex landscape of data management, having robust tools to monitor and analyze your data is paramount. InsightIQ 5.0.0 is your gateway to exploring the depths of historical data sourced from PowerScale OneFS. By leveraging its capabilities, you can monitor cluster activities, analyze performance, and gain insights to ensure optimal functionality.

Monitoring Clusters with Dynamic Dashboard

The InsightIQ Dashboard stands as a central hub for monitoring all your clusters, offering a comprehensive overview of their statuses and vital statistics. The dashboard facilitates quick interpretation of data and the action-based navigation links allow you to easily check on the observed statuses.

Here's a breakdown of the essential sections within this powerful monitoring interface:

Figure 1 IIQ Dashboard

InsightIQ Status

This section provides an overview of connected clusters, highlighting monitoring errors and any suspended monitoring activities. The InsightIQ Status icons offer a quick assessment of the monitored clusters: green signifies active monitoring, red indicates monitoring errors, grey denotes suspended monitoring or incomplete credentials. There is a fourth status icon. Blue indicates the number of PowerScale clusters whose status is outside of green, red, or grey status values. This is typically due to an internal error.

Additionally, the InsightIQ Datastore Usage icons provide insights into datastore health, with green indicating health, yellow signaling near-full capacity, and red alerting that the datastore has reached its maximum limit.

Alerts

The Alerts section within InsightIQ is a pivotal area displaying crucial data accumulated over the past 24 hours, categorized by severity—emergency, critical, warning, and information. This section shows the top three clusters with the highest number of alerts, granting immediate visibility into potential issues impacting PowerScale clusters. The dashboard offers a swift way to access the 'Alerts' section, where you have the capability to create alerts, defining Key Performance Indicators (KPIs) and thresholds, easily viewable on the dashboard for prompt action. This comprehensive alert system ensures timely responses to potential issues.

Aggregated Capacity for monitored clusters

Get insights into the used and free raw capacity across monitored clusters, as well as the estimated total usable capacity.

Performance Overview

This section presents average values for critical performance metrics like protocol latency, network throughput, CPU usage, protocol operations, active clients, and active jobs, displaying changes in statistics over the past 24 hours. It also offers a convenient link to navigate to the 'View Performance Report', facilitating in-depth analysis across various metrics.   

Monitored Clusters by % Used Capacity

This detailed breakdown showcases used capacity, free raw capacity, estimated usable capacity, and data reduction ratio for each monitored cluster. While it doesn't offer historical data, it provides real-time insights into the present cluster status. It offers quick navigation links to access both Capacity Reports and the Data Reduction Report for easy reference.   

Performance and File System Reports

The heart of InsightIQ lies in its ability to provide detailed performance reports and file system reports. These reports can be standardized or tailored to your specific needs, enabling you to track storage cluster performance efficiently. You also have the flexibility to generate Performance Reports as PDF files on a predefined schedule, enabling easy distribution via email attachments. 

InsightIQ reports are configured using modules, breakouts, and filter rules, providing a granular view of cluster components at specific data points. By employing modules and applying breakouts or filter rules, users can focus on distinct cluster components or specific attributes across the entire report. This flexibility allows the creation of tailored reports for various monitoring purposes.

Harnessing detailed metrics and insights empowers decision-making for crafting insightful performance reports. For instance, if network traffic surpasses anticipated levels across all monitored clusters, InsightIQ enables the creation of customized reports displaying detailed network throughput data. Analyzing direction-specific throughput assists in pinpointing any specific contribution to the overall traffic, aiding in precise troubleshooting and optimization strategies.

Figure 2 Sample Cluster Performance Report

Partitioned Performance

The Partition Performance report presents data from configurable datasets, offering insights into the top workloads consuming the most resources within a specific time range. Key data modules include: Dataset Summary, Workload Latency, Workload IOPS, Workload CPU Time, Workload Throughput, and Workload L2/L3 Cache Hits.

For more detailed information, users can focus on modules by average, top workload by max value, or pinned workload by average. 

Note: access to the Partition Performance Report, the InsightIQ user on the PowerScale cluster needs ISI_PRIV_PERFORMANCE with read permission. If unable to view the report, contact the InsightIQ admin or refer to the Dell Technologies InsightIQ 5.0.0 Administration Guide for permission configuration.

Figure 3  Sample Partitioned Performance Report

File System Analytic Report

File System Analytics (FSA) reports offer a comprehensive overview of the files stored within a cluster, providing essential insights into their types, locations, and usage.

InsightIQ supports two key FSA report categories: 

• Data Usage reports focus on individual file data, revealing details like unchanged file durations. 
• Data Property reports offer insights into the entire cluster file system, showcasing data such as file changes over specific periods and facilitating comparisons between different timeframes.

These reports help in understanding relative changes in file counts based on physical size, offering nuanced perspectives for effective file system management. For instance, by comparing Data Property reports of different clusters, you can observe patterns in file utilization—identifying clusters with regular file changes versus those housing less frequently modified files. Detecting inactive files through Data Usage reports facilitates efficient storage archiving strategies, optimizing cluster space.

These reports also play a pivotal role in verifying the expected behavior of cluster file systems. For example, dedicated archival clusters can be monitored using Data Property reports to observe file count changes. An unexpectedly high count might prompt storage admin to consider relocating files to development clusters, ensuring efficient resource utilization.

Figure 4 Data Properties Report

These FSA reports within InsightIQ not only provide visibility into cluster file systems but also serve as strategic tools for efficient storage management and troubleshooting unexpected discrepancies.

Conclusion: Empowering Data Management

InsightIQ isn't just a monitoring tool. It's a comprehensive suite offering a multitude of functionalities. It's about transforming data into actionable insights, enabling users to make informed decisions and stay ahead in the dynamic world of data management. The robust features, customizable reports, and analytics capabilities make it an invaluable asset for ensuring the optimal performance and health of PowerScale OneFS clusters.

Overview

InsightIQ 5.0.0 introduces two distinct deployment options catering to varying scalability needs: InsightIQ Simple and InsightIQ Scale. Let's delve into the overview of both offerings to guide your deployment decision-making process.

InsightIQ Simple

Designed for the straightforward deployment and moderate scalability, IIQ Simple accommodates up to 252 nodes or 10 clusters. Here's a snapshot of its key requirements:

• Target Use Case: Simple deployment scenarios with moderate scaling requirements.
• Deployment Method: VMware-based deployment using the OVA template.
• OS Requirements: ESXi 7.0.3 and 8.0.2.
• Hardware Requirements: VMware hardware version 15 or higher, requires a CPU of 12 vCPU, 32GB memory, and 1.5TB disk space. 

InsightIQ Scale

For organizations demanding extreme scalability, IIQ Scale steps in, supporting up to 504 nodes or 20 clusters, with potential expansion post IIQ 5.0. The details include:

• Target Use Case: Extreme scalability requirements.
• Deployment Method: RHEL-based deployment utilizing a specialized deployment script.
• OS Requirements: RHEL 8.6 x64.
• Hardware Requirements: Three virtual machines or physical servers, each with a configuration of 12 vCPU, 32GB memory, and specific storage options based on the chosen datastore location.

Here is the summary table:

Leveraging the NFS export

In InsightIQ deployments, leveraging an NFS export for the datastore, whether from a PowerScale cluster or a Linux NFS server, can significantly enhance scalability. However, to ensure a seamless setup, specific prerequisites must be addressed:

Access and Permissions:

• Guarantee accessibility of the NFS server and export from all servers/VMs where InsightIQ is deployed. This accessibility is crucial for uninterrupted data flow.
• Set read/write permissions (chmod 777 <export_path>) to ensure unrestricted access for all users utilizing the NFS export. 

Security Measures:

• Root User Mapping: Avoid mapping the root user (no_root_squash) to maintain secure access control.
• Mount Access: Enable mount access to subdirectories for streamlined data retrieval and utilization.

Resource Allocation:

• Allocate a substantial 1.5TB for the NFS export, ensuring ample space for data storage and future scalability.
• Allocate 200GB free space on the root partition ("/") of all servers/VMs hosting InsightIQ.

By ensuring compliance with these guidelines, organizations can unlock the full potential of InsightIQ while maintaining a robust and reliable infrastructure.

IIQ 5.0.0 Support Matrix

A concise summary detailing supported OneFS versions, host OS, recommended client display configurations, and browser compatibility for both IIQ Simple and IIQ Scale deployments.

InsightIQ uses TLS 1.3 exclusively. A web browser without TLS 1.3 enabled or supported cannot access InsightIQ 5.0.0.

Summary

Unlocking the full potential of InsightIQ 5.0.0 is a journey that begins with a clear understanding of deployment options and their prerequisites. Embracing your organization's scalability needs and infrastructure nuances ensures a seamless 

The Dell PowerScale CloudPools feature of OneFS allows tiering, enabling you to move cold or infrequently accessed data to lower-cost cloud storage. CloudPools extends the PowerScale namespace to the private cloud and the public cloud.

This blog focuses on CloudPools supported cloud providers.

CloudPools supported cloud providers

Each cloud provider offers a range of storage classes that you can choose from based on data access and cost requirements. CloudPools does not support some specific storage classes that may cause CloudPools operations failure due to unacceptable object read/write latency.

Table 1. Supported and unsupported cloud providers and storage classes for CloudPools

To address cost requirements, users can move objects from higher cost storage class to lower cost storage class on the cloud. The tiers are as follows:

• Supported/Tier1 storage classes: CloudPools supports the storage classes. 
• Tier2 storage classes:  Amazon S3 supports using S3 Lifecycle policy to move CloudPools objects from the Tier1 storage class to the Tier2 storage class. This movement will not break CloudPools operations.
• Tier3 storage classes: Amazon S3 supports using S3 Lifecycle policy to move CloudPools objects from the Tier1 storage class to the Tier3 storage class. This movement will break CloudPools operations. CloudPools objects must first be moved to Tier2 storage class or Tier1 storage class to be accessed.

Table 2. Storage classes per tier per cloud provider in CloudPools

[1] Assumes no opt-in to Deep Archive Access Tier.
[2] Assumes real-time access is enabled.
[3] S3 directory buckets only allow objects stored in the S3 Express One Zone storage class and do not support S3 Lifecycle policies.

Resources

• Dell PowerScale: CloudPools and ECS
• Dell PowerScale: CloudPools and Amazon Web Services
• Dell PowerScale: CloudPools and Microsoft Azure
• Dell PowerScale: CloudPools and Google Cloud
• Dell PowerScale: CloudPools and Alibaba Cloud

Author: Jason He, Principal Engineering Technologist

Backing up and restoring OneFS cluster configurations is not new, as it was introduced in OneFS 9.2. However, only a limited set of components can be backed up or restored. This is a popular feature and we have received a lot of feedback that we should add more supported components. Now, with the release of OneFS 9.7, this feature gets a big enhancement. The following is a complete list of the components we support in 9.7. (The new ones are marked in blue.)

Some other enhancements include:

1. Lock configuration during backup
2. Support custom rules for restoring subnet IP addresses

Next, I’ll walk you through an example and explain the details of these enhancements.

Let’s take a look at the backup first.

Like what we have in the previous version, backup and restore are only available through PAPI and CLI (there is no WebUI at this stage). But I can guarantee you that the overall process is very simple and straightforward. If you are familiar with how to do it in the previous version, it’s almost the same.

You can use the following CLI command to back up a cluster configuration:

isi cluster config exports create [--components …]

Here is an example where I want to export the network configuration:

# isi cluster config exports create –components=Network
The following components’ configuration are going to be exported:
[‘Network’]
Notice:
    The exported configuration will be saved in plain text. It is recommended to encrypt it according to your specific requirements.
Do you want to continue? (yes/[no]): yes
This may take a few seconds, please wait a moment
Created export task ‘vshen-0eis0wn-20231128032252’

You can see that once the backup is triggered, a task is automatically created, and you can use the following command to view the details of the task:

isi cluster config exports view <export-id>

Here is what I have in my environment:

# isi cluster config exports view –id vshen-0eis0wn-20231128032252
     ID: vshen-0eis0wn-20231128032252
 Status: Successful
   Done: [‘network’]
 Failed: []
Pending: []
Message:
   Path: /ifs/data/Isilon_Support/config_mgr/backup/vshen-0eis0wn-20231128032252

During backup, to make a consistent configuration, a temporary lock is enabled to prevent new PAPI calls like POST, PUT, and DELETE. (The GET method will not be impacted.) In most cases, the backup job is completed quickly and it releases the lock when it finishes running.

You can use the following command to view the backup lock:

# isi cluster config lock view
Configuration lock enabled: Yes

You can also use the CLI command to manually enable or disable the lock:

# isi cluster config lock modify –action=enable
WARNING: User won’t be able to make any configuration changes after enabling configuration lock.
Are you sure you want to enable configuration lock? (yes/[no]): yes

After the backup task completes, the backup files will be generated under: /ifs/data/Isilon_Support/config_mgr/backup. Although the backup files are in plain text format, the sensitive information doesn’t appear here.

cat ./network_vshen-0eis0wn-20231128032252.json
{
  "description": {
    "component": "network",
    "release": "9.7.0.0",
    "action": "backup",
    "job_id": "vshen-0eis0wn-20231128032252",
    "result": "successful",
    "errors": []
  },
  "network": {
    "dnscache": {
      "cache_entry_limit": 65536,
       "cluster_timeout": 5,
       "dns_timeout": 5,
       "eager_refresh": 0,
       "testping_delta": 30,
       "ttl_max_noerror": 3600,
       "ttl_max_nxdomain": 3600,
…

When doing an import, you can use a command similar to the following:

# isi cluster config imports create --export-id=vshen-0eis0wn-20231128032252
Source Cluster Information:
          Cluster name: vshen-0eis0wn
       Cluster version: 9.7.0.0
            Node count: 1
  Restoring components: ['network']
Notice:
    Please review above information and make sure the target cluster has the same hardware configuration as the source cluster, otherwise the restore may fail due to hardware incompatibility. Please DO NOT use or change the cluster while configurations are being restored. Concurrent modifications are not guaranteed to be retained and some data services may be affected.
Do you want to continue? (yes/[no]): yes
This may take a few seconds, please wait a moment
Created import task 'vshen-0eis0wn-20231128064821'

When you deal with network component restore, to avoid connectivity breaks you can restore the configuration without destroying any existing subnets or pools’ IP addresses.

To do this, use the parameter “--network-subnets-ip”:

# isi cluster config imports create --export-id=vshen-0eis0wn-20231128032252 --network-subnets-ip="groupnet0.subnet0:10.242.114.0/24"
Source Cluster Information:
          Cluster name: vshen-0eis0wn
       Cluster version: 9.7.0.0
            Node count: 1
  Restoring components: ['network']
Notice:
    Please review above information and make sure the target cluster has the same hardware configuration as the source cluster, otherwise the restore may fail due to hardware incompatibility. Please DO NOT use or change the cluster while configurations are being restored. Concurrent modifications are not guaranteed to be retained and some data services may be affected.
Do you want to continue? (yes/[no]): yes
This may take a few seconds, please wait a moment
Created import task 'vshen-0eis0wn-20231128070157'

That’s how it works! As I said, it’s very simple and straightforward. If you see any errors, you can check the log: /var/log/config_mgr.log.

Author: Vincent Shen

Dell PowerScale is already powering up the holiday season with the launch of the innovative OneFS 9.7 release, which shipped today (13^th December 2023). This new 9.7 release is an all-rounder, introducing PowerScale innovations in Cloud, Performance, Security, and ease of use.

After the debut of APEX File Storage for AWS earlier this year, OneFS 9.7 extends and simplifies the PowerScale in the public cloud offering, delivering more features on more instance types across more regions.

In addition to providing the same OneFS software platform on-prem and in the cloud, and customer-managed for full control, APEX File Storage for AWS in OneFS 9.7 sees a 60% capacity increase, providing linear capacity and performance scaling up to six SSD nodes and 1.6 PiB per namespace/cluster, and up to 10GB/s reads and 4GB/s writes per cluster. This can make it a solid fit for traditional file shares and home directories, vertical workloads like M&E, healthcare, life sciences, finserv, and next-gen AI, ML and analytics applications.

Enhancements to APEX File Storage for AWS

PowerScale’s scale-out architecture can be deployed on customer managed AWS EBS and ECS infrastructure, providing the scale and performance needed to run a variety of unstructured workflows in the public cloud. Plus, OneFS 9.7 provides an ‘easy button’ for streamlined AWS infrastructure provisioning and deployment.

Once in the cloud, you can further leverage existing PowerScale investments by accessing and orchestrating your data through the platform's multi-protocol access and APIs.

This includes the common OneFS control plane (CLI, WebUI, and platform API), and the same enterprise features: Multi-protocol, SnapshotIQ, SmartQuotas, Identity management, and so on.

With OneFS 9.7, APEX File Storage for AWS also sees the addition of support for HDFS and FTP protocols, in addition to NFS, SMB, and S3. Granular performance prioritization and throttling is also enabled with SmartQoS, allowing admins to configure limits on the maximum number of protocol operations that NFS, S3, SMB, or mixed protocol workloads can consume on an APEX File Storage for AWS cluster.

Security

With data integrity and protection being top of mind in this era of unprecedented cyber threats, OneFS 9.7 brings a bevy of new features and functionality to keep your unstructured data and workloads more secure than ever. These new OneFS 9.7 security enhancements help address US Federal and DoD mandates, such as FIPS 140-2 and DISA STIGs – in addition to general enterprise data security requirements. Included in the new OneFS 9.7 release is a simple cluster configuration backup and restore utility, address space layout randomization, and single sign-on (SSO) lookup enhancements.

Data mobility

On the data replication front, SmartSync sees the introduction of GCP as an object storage target in OneFS 9.7, in addition to ECS, AWS and Azure. The SmartSync data mover allows flexible data movement and copying, incremental resyncs, push and pull data transfer, and one-time file to object copy.

Performance improvements

Building on the streaming read performance delivered in a prior release, OneFS 9.7 also unlocks dramatic write performance enhancements, particularly for the all-flash NVMe platforms - plus infrastructure support for future node hardware platform generations. A sizable boost in throughput to a single client helps deliver performance for the most demanding GenAI workloads, particularly for the model training and inferencing phases. Additionally, the scale-out cluster architecture enables performance to scale linearly as GPUs are increased, allowing PowerScale to easily support AI workflows from small to large.

Cluster support for InsightIQ 5.0

The new InsightIQ 5.0 software expands PowerScale monitoring capabilities, including a new user interface, automated email alerts, and added security. InsightIQ 5.0 is available today for all existing and new PowerScale customers at no additional charge. These innovations are designed to simplify management, expand scale and security, and automate operations for PowerScale performance monitoring for AI, GenAI, and all other workloads.

In summary, OneFS 9.7 brings the following new features and functionality to the Dell PowerScale ecosystem:

We’ll be taking a deeper look at these new features and functionality in blog articles over the course of the next few weeks. 

Meanwhile, the new OneFS 9.7 code is available on the Dell Support site, as both an upgrade and reimage file, allowing both installation and upgrade of this new release. 

In this article, we’ll take a quick peek at the new PowerScale Hybrid H700/7000 and Archive A300/3000 hardware platforms that were released last month. So, the current PowerScale platform family hierarchy is as follows:

Here’s the lowdown on the new additions to the hardware portfolio: 

The PowerScale H700 provides performance and value to support demanding file workloads. With up to 960 TB of HDD per chassis, the H700 also includes inline compression and deduplication capabilities to further extend the usable capacity.

The PowerScale H7000 is a versatile, high performance, and high capacity hybrid platform with up to 1280 TB per chassis. The deep chassis based H7000 is an ideal to consolidate a range of file workloads on a single platform. The H7000 includes inline compression and deduplication capabilities.

On the active archive side, the PowerScale A300 combines performance, near-primary accessibility, value, and ease of use. The A300 provides between 120 TB to 960 TB per chassis and scales to 60 PB in a single cluster. The A300 includes inline compression and deduplication capabilities. 

PowerScale A3000: is an ideal solution for high performance, high density, and deep archive storage that safeguards data efficiently for long-term retention. The A3000 stores up to 1280 TB per chassis and scales to north of 80 PB in a single cluster. The A3000 also includes inline compression and deduplication.

These new H700/7000 and A300/3000 nodes require OneFS 9.2.1, and can be seamlessly added to an existing cluster. The benefits of offering the full complement of OneFS data services includes: snapshots, replication, quotas, analytics, data reduction, load balancing, and local and cloud tiering. All also contain SSD.

Unlike the all-flash PowerScale F900, F600, and F200 stand-alone nodes, which required a minimum of 3 nodes to form a cluster, a single chassis of 4 nodes is required to create a cluster which offers support for both InfiniBand and Ethernet backend network connectivity. 

Each F700/7000 and A300/3000 chassis contains four compute modules (one per node), and five drive containers, or sleds, per node. These sleds occupy bays in the front of each chassis, with a node’s drive sleds stacked vertically:

The drive sled is a tray which slides into the front of the chassis and contains between three and four 3.5 inch drives in an H700/0 or A300/0, depending on the drive size and configuration of the particular node. Both regular hard drives or self-encrypting drives (SEDs) are available in 2,4, 8, 12, and 16TB capacities.

Each drive sled has a white ‘not safe to remove’ LED on its front top left, as well as a blue power/activity LED, and an amber fault LED.

The compute modules for each node are housed in the rear of the chassis, and contain CPU, memory, networking, and SSDs, and power supplies. Nodes 1 & 2 are a node pair, as are nodes 3 & 4. Each node-pair shares a mirrored journal and two power supplies.

Here’s the detail of an individual compute module, which contains a multi core Cascade Lake CPU, memory, M2 flash journal, up to two SSDs for L3 cache, six DIMM channels, front end 40/100 or 10/25 Gb ethernet, 40/100 or 10/25 Gb ethernet or Infiniband, an ethernet management interface, and power supply and cooling fans:

Of particular interest is the ‘journal active’ LED, which is displayed as a white ‘hand icon’. When this is illuminated, it indicates that the mirrored journal is actively vaulting. 

A node’s compute module should not be removed from the chassis while this while LED is lit!

On the front of each chassis is an LCD front panel control with back-lit buttons and 4 LED Light Bar Segments - 1 per Node. These LEDs typically display blue for normal operation or yellow to indicate a node fault. This LCD display is hinged so it can be swung clear of the drive sleds for non-disruptive HDD replacement, for example.

So, in summary, the new Gen6 hardware delivers:

• More Power
  • More cores, more memory and more cache 
  • A300/3000 up to 2x faster than previous generation (A200/2000)
• More Choice
  • 100GbE, 25GbE and Infiniband options for cluster interconnect
  • Node compatibility for all hybrid and archive nodes
  • 30TB to 320TB per rack unit 
• More Value
  • Inline data reduction across the PowerScale family
  • Lowest $/GB and most density among comparable solutions

In the rapidly evolving landscape of technology, we find ourselves on the brink of a major technological leap with the integration of artificial intelligence (AI) into our daily lives. The potential impact of AI on the global economy is staggering, with forecasts predicting a whopping $13 trillion contribution. While the idea of AI isn't entirely new in the security sector which has previously employed analytics to monitor and report pixel changes in CCTV footage, the integration of AI technologies such as machine and deep learning has opened up a world of possibilities. One particularly rich source of data that organizations are eager to harness is video data, which is pivotal in a variety of use cases including operational improvements for retail, marketing strategies, and the enhancement of overall customer experiences.

Industries across the board are exploring AI's ability to enhance business efficiency, underscored by a whopping 63% of enterprise clients considering their security data as mission critical. That said, the success of AI deployments hinges on the collection and storage of data. AI models thrive on large, diverse datasets to achieve effectiveness and accuracy. For instance, when analyzing traffic patterns within a city, having access to comprehensive data spanning multiple seasons allows for more accurate planning. This necessity has led to the emergence of exceptionally large storage volumes to cater to AI's insatiable appetite for data.

A considerable portion of data – approximately 80% – collected by organizations is unstructured, including video data. Data scientists are faced with the arduous task of mapping this unstructured data into their models, thanks in part to the fragmented nature of security solutions. Shockingly, over 79% of a data scientists’ time is consumed by data wrangling and collection rather than actual data analysis due to siloed data storage. Complex scenarios involving thousands of cameras pointed at different targets further complicate the application of AI models to this data.

Recent discussions in the field of AI have introduced the concept of ‘Data Fuzion,’ which underscores the importance of consolidating and harmonizing data, overcoming the current infrastructure's obstacles, and making data more accessible and usable for data science applications in the security industry. There is a significant divide between the potential for data science solutions to drive business outcomes and the actual implementation, largely attributed to – as previously mentioned – the fragmented, siloed nature of data storage and the scarcity of in-house data science expertise.

The AI solutions available today in the security domain often come as black box offerings with pre-programmed models, however end-users are increasingly seeking low- or no-code AI tools that allow them to tailor and modify models to meet their specific organizational needs. This shift enables organizations to fine-tune AI to their precise requirements, further optimizing business outcomes. Additionally, the rise of cloud computing has presented budgetary challenges as organizations are increasingly paying for data access, leading to a trend of cloud repatriation – moving data back to on-premises environments to better manage costs and reduce latency in real-time applications.

AI is transforming the way organizations protect not only their external security but also their internal data. Dell Technologies, for example, offers a solution known as Ransomware Defender within its unstructured data offerings, an AI-based detection tool which identifies anomalies and takes action when malicious actors attempt to encrypt or delete data by modeling typical behaviors and sounding alarms when suspicious activities occur. Check out the Dell Technologies cyber security solution page for more information.

To fully harness the power of AI and navigate these complex data landscapes, organizations are turning to single-volume unstructured data solutions that embody the concept of ‘Data Fuzion.’ Dell Technologies Unstructured Data Solutions, with their petabyte-scale single-volume architecture, offer not only the ability to support this burgeoning workload but also robust cyber protection and multi-cloud capabilities. In this way, organizations can chart a seamless path towards AI adoption while ensuring data-driven security and efficiency. Visit the Dell Technologies PowerScale solutions page to learn more.

Resources

• Dell Security Services
• Dell PowerScale

Authors: Mordekhay Shushan | Safety and Security Solution Architect & Brian Stonge | Business Development Manager, Video Safety and Security

Earlier in this series, we took a look at the architecture of the new OneFS WebUI SSO functionality. Now, we move on to its management and troubleshooting.

As we saw in the previous article, once the IdP and SP are configured, a cluster admin can enable SSO per access zone using the OneFS WebUI by navigating to Access > Authentication providers > SSO. From here, select the desired access zone and click the ‘Enable SSO’ toggle:

Or from the OneFS CLI using the following syntax:

# isi auth sso settings modify --sso-enabled 1

Once complete, the SSO configuration can be verified from a client web browser by browsing to the OneFS login screen. If all is operating correctly, redirection to the ADFS login screen will occur. For example:

After successful authentication with ADFS, cluster access is granted and the browser session is redirected back to the OneFS WebUI .

In addition to the new SSO WebUI pages, OneFS 9.5 also adds a subcommand to the ‘isi auth’ command set for configuring SSO from the CLI. This new syntax includes:

• isi auth sso idps
• isi auth sso settings  
• isi auth sso sp

With these, you can use the following procedure to configure and enable SSO using the OneFS command line.

1. Define the ADFS instance in OneFS.

Enter the following command to create the IdP account:

# isi auth ads create <domain_name> <user> --password=<password> ...

where:

2. Next, add the IdP to the pertinent OneFS zone. Note that each of a cluster’s access zone(s) must have an IdP configured for it. The same IdP can be used for all the zones, but each access zone must be configured separately.

# isi zone zones modify --add-auth-providers

For example:

# isi zone zones modify system --add-auth-providers=lsa-activedirectoryprovider:idp1.isilon.com

3. Verify that OneFS can find users in Active Directory.

# isi auth users view idp1.isilon.com\\<username>

In the output, ensure that an email address is displayed. If not, return to Active Directory and assign email addresses to users.

4. Configure the OneFS hostname for SAML SSO.

# isi auth sso sp modify --hostname=<name>

Where <name> is the name that SAML SSO can use to represent the OneFS cluster to ADFS. SAML redirects clients to this hostname.

5. Obtain the ADFS metadata and store it under /ifs on the cluster.

In the following example, an HTTPS GET request is issued using the 'curl' utility to obtain the metadata from the IDP and store it under /ifs on the cluster.

# curl -o /ifs/adfs.xml https://idp1.isilon.com/FederationMetadata/2007-06/ FederationMetadata.xml

6. Create the IdP on OneFS using the ‘metadata-location’ path for the xml file in the previous step.

# isi auth sso idps create idp1.isilon.com --metadata-location="/ifs/adfs.xml"

7. Enable SSO:

# isi auth sso settings modify --sso-enabled=yes -–zone <zone>

Use the following syntax to view the IdP configuration:

# isi auth sso idps view <idp_ID>

For example:

# isi auth sso idps view idp
ID: idp
Metadata Location: /ifs/adfs.xml
Entity ID: https://dns.isilon.com/adfs/services/trust
Login endpoint of the IDP
URL: https://dns.isilon.com/adfs/ls/
Binding: wrn:oasis:names:tc:SAML:2.0:bidings:HTTP-Redirect
Logout endpoint of the IDP
URL: https://dns.isilon.com/adfs/ls/
Binding: wrn:oasis:names:tc:SAML:2.0:bidings:HTTP-Redirect
Response URL: -
Type: metadata
Signing Certificate: -
        Path:
        Issuer: CN-ADFS Signing – dns.isilon.com
        Subject: CN-ADFS Signing – dns.isilon.com
        Not Before: 2023-02-02T22:22:00
        Not After: 2024-02-02T22:22:00
        Status: valid
Value and Type
        Value: -----BEGIN CERTIFICATE-----
MITC9DCCAdygAwIBAgIQQQQc55appr1CtfPNj5kv+DANBgkqhk1G9w8BAQsFADA2
<snip>

Troubleshooting

If the IdP and/or SP Signing certificate happens to expire, users will be unable to login to the cluster with SSO and an error message will be displayed on the login screen.

In this example, the IdP certificate has expired, as described in the alert message. When this occurs, a warning is also displayed on the SSO Authentication page, as shown here:

To correct this, download either a new signing certificate from the identity provider or a new metadata file containing the IdP certificate details. When this is complete, you can then update the cluster’s IdP configuration by uploading the XML file or the new certificate.

Similarly, if the SP certificate has expired, the following notification alert is displayed upon attempted login:

The following error message is also displayed on the WebUI SSO tab, under Access > Authentication providers > SSO, along with a link to regenerate the metadata file:

The expired SP signing key and certificate can also be easily regenerated from the OneFS CLI:

# isi auth sso sp signing-key rekey

This command will delete any existing signing key and certificate and replace them with a newly generated signing key and certificate. Make sure the newly generated certificate is added to the IDP to ensure that the IDP can verify messages sent from the cluster. Are you sure?  (yes/[no]):   yes
# isi auth sso sp signing-key dump
-----BEGIN CERIFICATE-----
MIIE6TCCAtGgAwIBAgIJAP30nSyYUz/cMA0GCSqGSIb3DQEBCwUAMCYxJDAiBgNVBAMMG1Bvd2VyU2NhbGUgU0FNTCBTaWduaWSnIEtleTAeFw0yMjExMTUwMzU0NTFaFw0yMzExMTUwMzU0NTFaMCYxJDAiBgNVBAMMG1Bvd2VyU2NhbGUgU0FNTCBTaWduaWSnIEtleTCCAilwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMOOmYJ1aUuxvyH0nbUMurMbQubgtdpVBevy12D3qn+x7rgym8/v50da/4xpMmv/zbE0zJ0IVbWHZedibtQhLZ1qRSY/vBlaztU/nA90XQzXMnckzpcunOTG29SMO3x3Ud4*fqcP4sKhV
<snip>

When it is regenerated, either the XML file or certificate can be downloaded, and the cluster configuration updated by either metadata download or manual copy:

Finally, upload the SP details back to the identity provider.

For additional troubleshooting of OneFS SSO and authentication issues, there are some key log files to check. These include:

Author: Nick Trimbee

In the previous article in this series, we took a look at the new NFS locks and waiters reporting CLI command set and API endpoints. Next, we turn our attention to some additional context, caveats, and NFSv3 lock removal.

Before the NFS locking enhancements in OneFS 9.5, the legacy CLI commands were somewhat inefficient.  Their output also included other advisory domain locks such as SMB, which made the output more difficult to parse. The table below maps the new 9.5 CLI commands (and corresponding handlers) to the old NLM syntax.

Note that the isi_classic nfs locks and waiters CLI commands have also been deprecated in OneFS 9.5.

When upgrading to OneFS 9.5 or later from a prior release, the legacy platform API handlers continue to function through and post upgrade. Thus, any legacy scripts and automation are protected from this lock reporting deprecation. Additionally, while the new platform API handlers will work in during a rolling upgrade in mixed-mode, they will only return results for the nodes that have already been upgraded (‘high nodes’).

Be aware that the NFS locking CLI framework does not support partial responses. However, if a node is down or the cluster has a rolling upgrade in progress, the alternative is to query the equivalent platform API endpoint instead.

Performance-wise, on very large busy clusters, there is the possibility that the lock and waiter CLI commands’ output will be sluggish. In such instances, the --timeout flag can be used to increase the command timeout window. Output filtering can also be used to reduce number of locks reported.

When a lock is in a transition state, there is a chance that it may not have/report a version. In these instances, the Version field will be represented as —. For example:

# isi nfs locks list -v
Client: 1/TMECLI1:487722/10.22.10.250
Client ID: 487722351064074
LIN: 4295164422
Path: /ifs/locks/nfsv3/10.22.10.250_1
Lock Type: exclusive
Range: 0, 92233772036854775807
Created: 2023-08-18T08:03:52
Version: -
---------------------------------------------------------------
Total: 1

This behavior should be experienced very infrequently. However, if it is encountered, simply execute the CLI command again, and the lock version should be reported correctly.

When it comes to troubleshooting NFSv3/NLM issues, if an NFSv3 client is consistently experiencing NLM_DENIED or other lock management issues, this is often a result of incorrectly configured firewall rules. For example, take the following packet capture (PCAP) excerpt from an NFSv4 Linux client:

   21 08:50:42.173300992  10.22.10.100 → 10.22.10.200 NLM 106    V4 LOCK Reply (Call In 19) NLM_DENIED

Often, the assumption is that only the lockd or statd ports on the server side of the firewall need to be opened and that the client always makes that connection that way. However, this is not the case. Instead, the server will continually respond with a ‘let me get back to you’, then later reconnect to the client. As such, if the firewall blocks access to rcpbind on the client and/or lockd or statd on the client, connection failures will likely occur.

Occasionally, it does become necessary to remove NLM locks and waiters from the cluster. Traditionally, the isi_classic nfs clients rm command was used, however that command has limitations and is fully deprecated in OneFS 9.5 and later. Instead, the preferred method is to use the isi nfs nlm sessions CLI utility in conjunction with various other ancillary OneFS CLI commands to clear problematic locks and waiters. 

Note that the isi nfs nlm sessions CLI command, available in all current OneFS version, is Zone-Aware. The output formatting is seen in the output for the client holding the lock as it now shows the Zone ID number at the beginning. For example:

 4/tme-linux1/10.22.10.250 

This represents:

Zone ID 4 / Client tme-linux1 / IP address of cluster node holding the connection.

A basic procedure to remove NLM locks and waiters from a cluster is as follows: 
 
1. List the NFS locks and search for the pertinent filename. 

In OneFS 8.5 and later, the locks list can be filtered using the --path argument.

# isi nfs locks list --path=<path> | grep <filename>

Be aware that the full path must be specified, starting with /ifs. There is no partial matching or substitution for paths in this command set.

For OneFS 9.4 and earlier, the following CLI syntax can be used:

#  isi_for_array -sX 'isi nfs nlm locks list | grep <filename>'

2. List the lock waiters associated with the same filename using |grep.

For OneFS 8.5 and later, the waiters list can also be filtered using the --path syntax:

# isi nfs locks waiters –path=<path> | grep <filename> 

With OneFS 9.4 and earlier, the following CLI syntax can be used:

# isi_for_array -sX 'isi nfs nlm locks waiters |grep -i <filename>'

3. Confirm the client and logical inode number (LIN) being waited upon. 

This can be accomplished by querying the efs.advlock.failover.lock_waiters sysctrl. For example:

# isi_for_array -sX 'sysctl efs.advlock.failover.lock_waiters'

[truncated output]
 ...
 client = { '4/tme-linux1/10.20.10.200’, 0x26593d37370041 }
 ...
resource = 2:df86:0218

Note that for sanity checking, the isi get -L CLI utility can be used to confirm the path of a file from its LIN:

isi get -L <LIN>

4. Remove the unwanted locks which are causing waiters to stack up. 

Keep in mind that the isi nfs nlm sessions command syntax is access zone-aware.

List the access zones by their IDs.

# isi zone zones list -v | grep -iE "Zone ID|name"

Once the desired zone ID has been determined, the isi_run -z CLI utility can be used to specify the appropriate zone in which to run the isi nfs nlm sessions commands: 

# isi_run -z 4 -l root

Next, the isi nfs nlm sessions delete CLI command will remove the specific lock waiter which is causing the issue. The command syntax requires specifying the client hostname and node IP of the node holding the lock. 

# isi nfs nlm sessions delete –-zone <AZ_zone_ID> <hostname> <cluster-ip>

For example:

# isi nfs nlm sessions delete –zone 4 tme-linux1 10.20.10.200
 Are you sure you want to delete all NFSv3 locks associated with client tme-linux1 against cluster IP 10.20.10.100? (yes/[no]): yes

5. Repeat the commands in step 1 to confirm that the desired NLM locks and waiters have been successfully culled.
 

BEFORE applying the process....

 # isi_for_array -sX 'isi nfs nlm locks list |grep JUN'
 TME-1: 4/tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_27JUN2017
 TME-1: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_28JUN2017
 TME-2: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_27JUN2017
 TME-2: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_28JUN2017
 TME-3: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_27JUN2017
 TME-3: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_28JUN2017
 TME-4: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_27JUN2017
 TME-4: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_28JUN2017
 TME-5: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_27JUN2017
 TME-5: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_28JUN2017
 TME-6: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_27JUN2017
 TME-6: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_28JUN2017
 
 
 # isi_for_array -sX 'isi nfs nlm locks waiters |grep -i JUN'
 TME-1: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_28JUN2017
 TME-1: 4/ tme-linux1/192.168.2.214  /ifs/tmp/TME/sequences/mncr_fabjob_seq_file_28JUN2017
 TME-2 exited with status 1
 TME-3 exited with status 1
 TME-4 exited with status 1
 TME-5 exited with status 1
 TME-6 exited with status 1

AFTER...

TME-1# isi nfs nlm sessions delete --hostname= tme-linux1 --cluster-ip=192.168.2.214
 Are you sure you want to delete all NFSv3 locks associated with client tme-linux1 against cluster IP 192.168.2.214? (yes/[no]): yes
 TME-1#
 TME-1#
 TME-1# isi_for_array -sX 'sysctl efs.advlock.failover.locks |grep 2:ce75:0319'
 TME-1 exited with status 1
 TME-2 exited with status 1
 TME-3 exited with status 1
 TME-4 exited with status 1
 TME-5 exited with status 1
 TME-6 exited with status 1
 TME-1#
 TME-1# isi_for_array -sX 'isi nfs nlm locks list |grep -i JUN'
 TME-1 exited with status 1
 TME-2 exited with status 1
 TME-3 exited with status 1
 TME-4 exited with status 1
 TME-5 exited with status 1
 TME-6 exited with status 1
 TME-1#
 TME-1# isi_for_array -sX 'isi nfs nlm locks waiters |grep -i JUN'
 TME-1 exited with status 1
 TME-2 exited with status 1
 TME-3 exited with status 1
 TME-4 exited with status 1
 TME-5 exited with status 1
 TME-6 exited with status 1

Author: Nick Trimbee

Included among the plethora of OneFS 9.5 enhancements is an updated NFS lock reporting infrastructure, command set, and corresponding platform API endpoints. This new functionality includes enhanced listing and filtering options for both locks and waiters, based on NFS major version, client, LIN, path, creation time, etc. But first, some backstory.

The ubiquitous NFS protocol underwent some fundamental architectural changes between its versions 3 and 4. One of the major differences concerns the area of file locking.

NFSv4 is the most current major version of the protocol, natively incorporating file locking and thereby avoiding the need for any additional (and convoluted) RPC callback mechanisms necessary with prior NFS versions. With NFSv4, locking is built into the main file protocol and supports new lock types, such as range locks, share reservations, and delegations/oplocks, which emulate those found in Window and SMB.

File lock state is maintained at the server under a lease-based model. A server defines a single lease period for all states held by an NFS client. If the client does not renew its lease within the defined period, all states associated with the client's lease may be released by the server. If released, the client may either explicitly renew its lease or simply issue a read request or other associated operation. Additionally, with NFSv4, a client can elect whether to lock the entire file or a byte range within a file. 

In contrast to NFSv4, the NFSv3 protocol is stateless and does not natively support file locking. Instead, the ancillary Network Lock Manager (NLM) protocol supplies the locking layer. Since file locking is inherently stateful, NLM itself is considered stateful. For example, when an NFSv3 filesystem mounted on an NFS client receives a request to lock a file, it generates an NLM remote procedure call instead of an NFS remote procedure call. 

The NLM protocol itself consists of remote procedure calls that emulate the standard UNIX file control (fcntl) arguments and outputs. Because a process blocks waiting for a lock that conflicts with another lock holder – also known as a ‘blocking lock’ – the NLM protocol has the notion of callbacks from the file server to the NLM client to notify that a lock is available. As such, the NLM client sometimes acts as an RPC server in order to receive delayed results from lock calls. 

Since NFSv3 is stateless, it requires more complexity to recover from failures like client and server outages and network partitions. If an NLM server crashes, NLM clients that are holding locks must reestablish them on the server when it restarts. The NLM protocol deals with this by having the status monitor on the server send a notification message to the status monitor of each NLM client that was holding locks. The initial period after a server restart is known as the grace period, during which only requests to reestablish locks are granted. Thus, clients that reestablish locks during the grace period are guaranteed to not lose their locks. 

When an NLM client crashes, ideally any locks it was holding at the time are removed from the pertinent NLM server(s). The NLM protocol handles this by having the status monitor on the client send a message to each server's status monitor once the client reboots. The client reboot indication informs the server that the client no longer requires its locks. However, if the client crashes and fails to reboot, the client's locks will persist indefinitely. This is undesirable for two primary reasons: Resources are indefinitely leaked. Eventually, another client will want to get a conflicting lock on at least one of the files the crashed client had locked and, as a result, the other client is postponed indefinitely.

Therefore, having NFS server utilities to swiftly and accurately report on lock and waiter status and utilities to clear NFS lock waiters is highly desirable for administrators – particularly on clustered storage architectures.

Prior to OneFS 9.5, the old NFS locking CLI commands were somewhat inefficient and also showed other advisory domain locks, which rendered the output somewhat confusing. The following table shows the new CLI commands (and corresponding handlers) which replace the older NLM syntax.

In OneFS 9.5 and later, the old API handlers will still exist to avoid breaking existing scripts and automation, however the CLI command syntax is deprecated and will no longer work.

Also be aware that the isi_classic nfs locks and waiters CLI commands have also been disabled in OneFS 9.5. Attempts to run these will yield the following warning message: 

# isi_classic nfs locks
This command has been disabled. Please use isi nfs for this functionality.

The new isi nfs locks CLI command output includes the following locks object fields:

Note that the ISI_NFS_PRIV RBAC privilege is required in order to view the NFS locks or waiters via the CLI or PAPI. In addition to ‘root’, the cluster’s ‘SystemAdmin’ and ‘SecurityAdmin’ roles contain this privilege by default.

Additionally, the new locks CLI command sets have a default timeout of 60 seconds. If the cluster is very large, the timeout may need to be increased for the CLI command. For example:

# isi –timeout <timeout value> nfs locks list

 The basic architecture of the enhanced NFS locks reporting framework is as follows:

The new API handlers leverage the platform API proxy, yielding increased performance over the legacy handlers. Additionally, updated syscalls have been implemented to facilitate filtering by NFS service and major version.

Since NFSv3 is stateless, the cluster does not know when a client has lost its state unless it reconnects. For maximum safety, the OneFS locking framework (lk) holds locks forever. The isi nfs nlm sessions CLI command allows administrators to manually free NFSv3 locks in such cases, and this command remains available in OneFS 9.5 as well as prior versions. NFSv3 locks may also be leaked on delete, since a valid inode is required for lock operations. As such, lkf has a lock reaper which periodically checks for locks associated with deleted files.

In OneFS 9.5 and later, current NFS locks can be viewed with the new isi nfs locks list command. This command set also provides a variety of options to limit and format the display output. In its basic form, this command generates a basic list of client IP address and the path. For example:

# isi nfs locks list
Client                              Path
-------------------------------------------------------------------
1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_1
1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_2
Linux NFSv4.0 TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv4/10.22.10.250_1
Linux NFSv4.0 TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv4/10.22.10.250_2
-------------------------------------------------------------------
Total: 4

To include more information, the -v flag can be used to generate a verbose locks listing:

 # isi nfs locks list -v
Client: 1/TMECLI1:487722/10.22.10.250
Client ID: 487722351064074
LIN: 4295164422
Path: /ifs/locks/nfsv3/10.22.10.250_1
Lock Type: exclusive
Range: 0, 92233772036854775807
Created: 2023-08-18T08:03:52
Version: v3
---------------------------------------------------------------
Client: 1/TMECLI1:487722/10.22.10.250
Client ID: 5175867327774721
LIN: 42950335042
Path: /ifs/locks/nfsv3/10.22.10.250_1
Lock Type: exclusive
Range: 0, 92233772036854775807
Created: 2023-08-18T08:10:31
Version: v3
---------------------------------------------------------------
Client: Linux NFSv4.0 TMECLI1:487722/10.22.10.250
Client ID: 487722351064074
LIN: 429516442
Path: /ifs/locks/nfsv3/10.22.10.250_1
Lock Type: exclusive
Range: 0, 92233772036854775807
Created: 2023-08-18T08:19:48
Version: v4
---------------------------------------------------------------
Client: Linux NFSv4.0 TMECLI1:487722/10.22.10.250
Client ID: 487722351064074
LIN: 4295426674
Path: /ifs/locks/nfsv3/10.22.10.250_2
Lock Type: exclusive
Range: 0, 92233772036854775807
Created: 2023-08-18T08:17:02
Version: v4
---------------------------------------------------------------
Total: 4

The previous syntax returns more detailed information for each lock, including client ID, LIN, path, lock type, range, created date, and NFS version.

The lock listings can also be filtered by client or client-id. Note that the --client option must be the full name in quotes:

# isi nfs locks list --client="full_name_of_client/IP_address" -v

For example:

# isi nfs locks list --client="1/TMECLI1:487722/10.22.10.250" -v
Client: 1/TMECLI1:487722/10.22.10.250
Client ID: 5175867327774721
LIN: 42950335042
Path: /ifs/locks/nfsv3/10.22.10.250_1
Lock Type: exclusive
Range: 0, 92233772036854775807
Created: 2023-08-18T08:10:31
Version: v3

Additionally, be aware that the CLI does not support partial names, so the full name of the client must be specified.

Filtering by NFS version can be helpful when attempting to narrow down which client has a lock. For example, to show just the NFSv3 locks:

# isi nfs locks list --version=v3 
Client                              Path
-------------------------------------------------------------------
1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_1
1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_2
-------------------------------------------------------------------
Total: 2

Note that the –-version flag supports both v3 and nlm as arguments and will return the same v3 output in either case. For example:

# isi nfs locks list --version=nlm
Client                              Path
-------------------------------------------------------------------
1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_1
1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_2
-------------------------------------------------------------------
Total: 2

Filtering by LIN or path is also supported. For example, to filter by LIN:

# isi nfs locks list --lin=42950335042 -v
Client: 1/TMECLI1:487722/10.22.10.250
Client ID: 5175867327774721
LIN: 42950335042
Path: /ifs/locks/nfsv3/10.22.10.250_1
Lock Type: exclusive
Range: 0, 92233772036854775807
Created: 2023-08-18T08:10:31
Version: v3

Or by path:

# isi nfs locks list --path=/ifs/locks/nfsv3/10.22.10.250_2
 -v
Client: Linux NFSv4.0 TMECLI1:487722/10.22.10.250
Client ID: 487722351064074
LIN: 4295426674
Path: /ifs/locks/nfsv3/10.22.10.250_2
Lock Type: exclusive
Range: 0, 92233772036854775807
Created: 2023-08-18T08:17:02
Version: v4

Be aware that the full path must be specified, starting with /ifs. There is no partial matching or substitution for paths in this command set.

Filtering can also be performed by creation time, for example:

# isi nfs locks list --created=2023-08-17T09:30:00 -v 

Note that when filtering by created, the output will include all locks that were created before or at the time provided.

The —limits argument can be used to curtail the number of results returned, and limits can be used in conjunction with all other query options. For example, to limit the output of the NFSv4 locks listing to one lock:

# isi nfs locks list -–version=v4 --limit=1 

Note that limit can be used with the range of query types.

The filter options are mutually exclusive with the exception of version. Note that version can be used with any of the other filter options. For example, filtering by both created and version.

This can be helpful when troubleshooting and trying to narrow down results.

In addition to locks, OneFS 9.5 also provides the isi nfs locks waiters CLI command set. Note that waiters are specific to NFSv3 clients, and the CLI reports any v3 locks that are pending and not yet granted.

Since NFSv3 is stateless, a cluster does not know when a client has lost its state unless it reconnects. For maximum safety, lk holds locks forever. The isi nfs nlm command allows administrators to manually free locks in such cases. Locks may also be leaked on delete, since a valid inode is required for lock operations. Thus, lkf has a lock reaper which periodically checks for locks associated with deleted files:

# isi nfs locks waiters

The waiters CLI syntax uses a similar range of query arguments as the isi nfs locks list command set.

In addition to the CLI, the platform API can also be used to query both NFS locks and NFSv3 waiters. For example, using curl to view the waiters via the OneFS pAPI:

# curl -k -u <username>:<passwd> https://localhost:8080/platform/protocols/nfs/waiters”
{
“total” : 2,
“waiters”;
}
{
“client” : “1/TMECLI1487722/10.22.10.250”,
“client_id” : “4894369235106074”,
“created” : “1668146840”,
“id” : “1 1YUIAEIHVDGghSCHGRFHTiytr3u243567klj212-MANJKJHTTy1u23434yui-ouih23ui4yusdftyuySTDGJSDHVHGDRFhgfu234447g4bZHXhiuhsdm”,
“lin” : “4295164422”,
“lock_type” : “exclusive”
“path” : “/ifs/locks/nfsv3/10.22.10.250_1”
“range” : [0, 92233772036854775807 ],
“version” : “v3”
}
},
“total” : 1
}

Similarly, using the platform API to show locks filtered by client ID:

# curl -k -u <username>:<passwd> “https://<address>:8080/platform/protocols/nfs/locks?client=<client_ID>”

For example:

# curl -k -u <username>:<passwd> “https://localhost:8080/platform/protocols/nfs/locks?client=1/TMECLI1487722/10.22.10.250”
{
“locks”;
}
{
“client” : “1/TMECLI1487722/10.22.10.250”,
“client_id” : “487722351064074”,
“created” : “1668146840”,
“id” : “1 1YUIAEIHVDGghSCHGRFHTiytr3u243567FCUJHBKD34NMDagNLKYGHKHGKjhklj212-MANJKJHTTy1u23434yui-ouih23ui4yusdftyuySTDGJSDHVHGDRFhgfu234447g4bZHXhiuhsdm”,
“lin” : “4295164422”,
“lock_type” : “exclusive”
“path” : “/ifs/locks/nfsv3/10.22.10.250_1”
“range” : [0, 92233772036854775807 ],
“version” : “v3”
}
},
“Total” : 1
}

Note that, as with the CLI, the platform API does not support partial name matches, so the full name of the client must be specified.

Author: Nick Trimbee

In the initial article in this series, we took a look at the OneFS SSL architecture, plus the first two steps in the basic certificate renewal or creation flow detailed below:

The following procedure includes options to complete a self-signed certificate replacement or renewal or to request an SSL replacement or renewal from a Certificate Authority (CA).

Signing the SSL Certificate

At this point, depending on the security requirements of the environment, the certificate can either be self-signed or signed by a Certificate Authority.

Self-Sign the SSL Certificate 

The following CLI syntax can be used to self-sign the certificate with the key, creating a new signed certificate which, in this instance, is valid for 1 year (365 days):     

# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

To verify that the key matches the certificate, ensure that the output of the following CLI commands return the same md5 checksum value:    

# openssl x509 -noout -modulus -in server.crt | openssl md5           

# openssl rsa -noout -modulus -in server.key | openssl md5

Next, proceed to the Add certificate to cluster section of this article once this step is complete. 

Use a CA to Sign the Certificate

If a CA is signing the certificate, ensure that the new SSL certificate is in x509 format and includes the entire certificate trust chain.

Note that the CA may return the new SSL certificate, the intermediate cert, and the root cert in different files. If this is the case, the PEM formatted certificate will need to be created manually.

Notably, the correct ordering is important when creating the PEM-formatted certificate. The SSL cert must be the top of the file, followed by the intermediate certificate, with the root certificate at the bottom. For example:        

-----BEGIN CERTIFICATE-----

<Contents of new SSL certificate>

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

<Contents of intermediate certificate>

<Repeat as necessary for every intermediate certificate provided by your CA>

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

<Contents of root certificate file>

-----END CERTIFICATE-----

A simple method for creating the PEM formatted file from the CLI is to cat them in the correct order as follows:

# cat CA_signed.crt intermediate.crt root.crt > onefs_pem_formatted.crt

Copy the onefs_pem_formatted.crt file to /ifs/tmp and rename it to server.crt. 

Note that if any of the aforementioned files are generated with a .cer extension, they should be renamed with a .crt extension instead.

The attributes and integrity of the certificate can be sanity checked with the following CLI syntax:       

# openssl x509 -text -noout -in server.crt

Adding the certificate to the cluster    

The first step in adding the certificate involves importing the new certificate and key into the cluster:      

# isi certificate server import /ifs/tmp/server.crt /ifs/tmp/server.key

Next, verify that the certificate imported successfully:     

# isi certificate server list -v 

The following CLI command can be used to show the names and corresponding IDs of the certificates:

# isi certificate server list -v | grep -A1 "ID:"

Set the imported certificate as default:      

# isi certificate settings modify --default-https-certificate=<id_of_cert_to_set_as_default>

Confirm that the imported certificate is being used as default by verifying status of Default HTTPS Certificate:     

# isi certificate settings view

If there is an unused or outdated cert, it can be deleted with the following CLI syntax:      

# isi certificate server delete --id=<id_of_cert_to_delete>

Next, view the new imported cert with command:      

# isi certificate server view --id=<id_of_cert>

Note that ports 8081 and 8083 still use the certificate from the local directory for SSL. Follow the steps below if you want to use the new certificates for port 8081/8083:

# isi services -a isi_webui disable
# chmod 640 server.key
# chmod 640 server.crt
# isi_for_array -s 'cp /ifs/tmp/server.key /usr/local/apache2/conf/ssl.key/server.key'
# isi_for_array -s 'cp /ifs/tmp/server.crt /usr/local/apache2/conf/ssl.crt/server.crt'
isi services -a isi_webui enable

Verifying the SSL certificate

There are two methods for verifying the updated SSL certificate.:

• Via the CLI, using the openssl command as follows:

# echo QUIT | openssl s_client -connect localhost:8080

• Or via a web browser, using the following URL:

https://<cluster_name>:8080

Note that where <cluster_name> is the FQDN or IP address, that’s typically used to access the cluster’s WebUI interface. The security details for the web page will contain the location and contact info, as above.

In both cases, the output includes location and contact info. For example:      

Subject: C=US, ST=<yourstate>, L=<yourcity>, O=<yourcompany>, CN=isilon.example.com/emailAddress=tme@isilon.com

Additionally, OneFS provides warning of an impending certificate expiry by sending a CELOG event alert, similar to the following:

SW_CERTIFICATE_EXPIRING: X.509 certificate default is nearing expiration: 
 
Event: 400170001
Certificate 'default' in '**' store is nearing expiration: 

Note that OneFS does not attempt to automatically renew a certificate. Instead, an expiring cert has to be renewed manually, per the procedure described above.

When adding an additional certificate, the matching cert is used any time you connect to that SmartConnect name via HTTPS. If no matching certificate is found, OneFS will automatically revert to using the default self-signed certificate.

Author: Nick Trimbee 

When using either the OneFS WebUI or platform API (pAPI), all communication sessions are encrypted using SSL (Secure Sockets Layer), also known as Transport Layer Security (TLS). In this series, we will look at how to replace or renew the SSL certificate for the OneFS WebUI.

SSL requires a certificate that serves two principal functions: It grants permission to use encrypted communication using Public Key Infrastructure and authenticates the identity of the certificate’s holder.

Architecturally, SSL consists of four fundamental components:

These sit in the stack as follows:

The basic handshake process begins with a client requesting an HTTPS WebUI session to the cluster. OneFS then returns the SSL certificate and public key. The client creates a session key, encrypted with the public key it is received from OneFS. At this point, the client only knows the session key. The client now sends its encrypted session key to the cluster, which decrypts it with the private key. Now, both the client and OneFS know the session key. So, finally, the session, encrypted using a symmetric session key, can be established. OneFS automatically defaults to the best supported version of SSL, based on the client request.

A PowerScale cluster initially contains a self-signed certificate, which can be used as-is or replaced with a third-party certificate authority (CA)-issued certificate. If the self-signed certificate is used upon expiry, it must be replaced with either a third-party (public or private) CA-issued certificate or another self-signed certificate that is generated on the cluster. The following are the default locations for the server.crt and server.key files.

The ‘isi certificate settings view’ CLI command displays all of the certificate-related configuration options. For example:

The above ‘certificate monitor enabled’ and ‘certificate pre expiration threshold’ configuration options govern a nightly cron job, which monitors the expiration of each managed certificate and fires a CELOG alert if a certificate is set to expire within the configured threshold. Note that the default expiration is 30 days (4W2D, which represents 4 weeks plus 2 days). The ‘ID: default’ configuration option indicates that this certificate is the default TLS certificate.

The basic certificate renewal or creation flow is as follows:

The steps below include options to complete a self-signed certificate replacement or renewal, or to request an SSL replacement or renewal from a Certificate Authority (CA).

Backing up the existing SSL certificate

The first task is to obtain the list of certificates by running the following CLI command, and identify the appropriate one to renew:

It’s always a prudent practice to save a backup of the original certificate and key. This can be easily accomplished using the following CLI commands, which, in this case, create the directory ‘/ifs/data/ssl_bkup’ directory, set the perms to root-only access, and copy the original key and certificate to it:

Renewing or creating a certificate

The next step in the process involves either the renewal of an existing certificate or creation of a certificate from scratch. In either case, first, create a temporary directory, for example /ifs/tmp:

a)       Renew an existing self-signed Certificate.

The following syntax creates a renewal certificate based on the existing ssl.key. The value of the ‘-days’ parameter can be adjusted to generate a certificate with the wanted expiration date. For example, the following command will create a one-year certificate.

Answer the system prompts to complete the self-signed SSL certificate generation process, entering the pertinent information location and contact information. For example:

Country Name (2 letter code) [AU]:US
 State or Province Name (full name) [Some-State]:Washington
 Locality Name (eg, city) []:Seattle
 Organization Name (eg, company) [Internet Widgits Pty Ltd]:Isilon
 Organizational Unit Name (eg, section) []:TME
 Common Name (e.g. server FQDN or YOUR name) []:isilon.com
 Email Address []:tme@isilon.com

When all the information has been successfully entered, the server.csr and server.key files will be generated under the /ifs/tmp directory.

Optionally,  the attributes and integrity of the certificate can be verified with the following syntax:

Next, proceed directly to the ‘Add the certificate to the cluster’ steps in section 4 of this article.

b)      Alternatively, a certificate and key can be generated from scratch, if preferred.

The following CLI command can be used to create an 2048-bit RSA private key:

Next, create a certificate signing request:

For example: 

Answer the system prompts to complete the self-signed SSL certificate generation process, entering the pertinent information location and contact information. Additionally, a ‘challenge password’ with a minimum of 4-bytes in length will need to be selected and entered.

As prompted, enter the information to be incorporated into the certificate request. When completed, the server.csr and server.key files will appear in the /ifs/tmp directory.

If wanted, a CSR file for a Certificate Authority, which includes Subject-Alternative-Names (SAN) can be generated. For example, additional host name entries can be added using a comma (IE. DNS:isilon.com,DNS:www.isilon.com).

In the next article, we will look at the certificate singing, addition, and verification steps of the process.

As on-the-wire encryption becomes increasingly commonplace, and often mandated via regulatory compliance security requirements, the policies applied in enterprise networks are rapidly shifting towards fully encrypting all traffic.

The OneFS SMB protocol implementation (lwio) has supported encryption for Windows and other SMB client connections to a PowerScale cluster since OneFS 8.1.1.

However, prior to OneFS 9.5, this did not include encrypted communications between the SMB redirector and Active Directory (AD) domain controller (DC). While Microsoft added support for SMB encryption in SMB 3.0, the redirector in OneFS 9.4 and prior releases only supported Microsoft’s earlier SMB 2.002 dialect.

When OneFS connects to Active Directory for tasks requiring remote procedure calls (RPCs), such as joining a domain, NTLM authentication, or resolving usernames and SIDs, these SMB connections are established from OneFS as the client connecting to a domain controller server.

As outlined in the Windows SMB security documentation, by default, and starting with Windows 2012 R2, domain admins can choose to encrypt access to a file share, which can include a domain controller. When encryption is enabled, only SMB3 connections are permitted.

With OneFS 9.5, the OneFS SMB redirector now supports SMB3, thereby allowing the Local Security Authority Subsystem Service (LSASS) daemon to communicate with domain controllers running Windows Server 2012 R2 and later over an encrypted session.

The OneFS redirector, also known as the ‘rdr driver’, is a stripped-down SMB client with minimal functionality, only supporting what is absolutely necessary.

Under the hood, OneFS SMB encryption and decryption use standard OpenSSL functions, and AES-128-CCM encryption is negotiated during SMB negotiate phase.

Although everything stems from the NTLM authentication requested by SMB server, the sequence of calls leads to the redirector establishing an SMB connection to the AD domain controller.

With OneFS 9.5, no configuration is required to enable SMB encryption in most situations, and there are no WebUI OR CLI configuration settings for the redirector.

With the default OneFS configuration, the redirector supports encryption if negotiated but it does not require it. Similarly, if the Active Directory domain requires encryption, the OneFS redirector will automatically enable and use encryption. However, if the OneFS redirector is explicitly configured to require encryption and the domain controller does not support encryption, the connection will fail.

The OneFS redirector encryption settings include:

The above keys and values are stored in the OneFS Likewise SMB registry and can be viewed and configured with the ‘lwreqshell’ utility. For example, to view the SMB redirector encryption config settings:

# /usr/likewise/bin/lwregshell list_values "HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr" | grep -i encrypt

    "Smb3EncryptionEnabled"   REG_DWORD       0x00000001 (1)

    "Smb3EncryptionRequired" REG_DWORD       0x00000000 (0)

The following syntax can be used to disable the ‘Smb3EncryptionRequired’ parameter by setting it to value ‘1’:

# /usr/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]" "Smb3EncryptionRequired" "0x00000001"

# /usr/likewise/bin/lwregshell list_values "HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr" | grep -i encrypt

    "Smb3EncryptionEnabled"   REG_DWORD       0x00000001 (1)

   "Smb3EncryptionRequired" REG_DWORD       0x00000001 (1)

Similarly, to restore the ‘Smb3EncryptionRequired’ parameter’s default value of ‘0’ (ie. not required):

# /usr/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]" "Smb3EncryptionEnabled" "0x00000001"

Note that, during the upgrade to OneFS 9.5, any nodes still running the old version will not be able to NTLM-authenticate if the DC they have affinity with requires encryption.

While redirector encryption is implemented in user space (in contrast to the SMB server, which is in the kernel), since it involves OpenSSL, the library does take advantage of hardware acceleration on the processor and utilizes AES-NI. As such, performance is only minimally impacted when the number of NTLM authentications to the AD domain is very large.

Also note that redirector encryption also only currently supports only AES-128-CCM encryption provided in the SMB 3.0.0 and 3.0.2 dialects. OneFS does not use AES-128-GCM encryption, available in the SMB 3.1.1 dialect (the latest), at this time.

When it comes to troubleshooting the redirector, the lwregshell tool can be used to verify its configuration settings. For example, to view the redirector encryption settings:

# /usr/likewise/bin/lwregshell list_values "HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr" | grep -i encrypt

    "Smb3EncryptionEnabled"   REG_DWORD       0x00000001 (1)

    "Smb3EncryptionRequired" REG_DWORD       0x00000000 (0)

Similarly, to find the maximum SMB version supported by the redirector:

# /usr/likewise/bin/lwregshell list_values "HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr" | grep -i dialect

    "MaxSmb2DialectVersion"   REG_SZ          "max"

The ‘lwsm’ CLI utility with the following syntax will confirm the status of the various lsass components:

# /usr/likewise/bin/lwsm list | grep lsass

lsass                       [service]     running (lsass: 5164)

netlogon                    [service]     running (lsass: 5164)

rdr                         [driver]      running (lsass: 5164)

It can also be used to show and modify the logging level. For example:

# /usr/likewise/bin/lwsm get-log rdr

<default>: syslog LOG_CIFS at WARNING

# /usr/likewise/bin/lwsm set-log-level rdr - debug

# /usr/likewise/bin/lwsm get-log rdr

<default>: syslog LOG_CIFS at DEBUG

When finished, rdr logging can be returned to its previous log level as follows:

# /usr/likewise/bin/lwsm set-log-level rdr - warning

# /usr/likewise/bin/lwsm get-log rdr

<default>: syslog LOG_CIFS at WARNING

Additionally, the existing ‘lwio-tool’ utility has been modified in OneFS 9.5 to include functionality allowing simple test connections to domain controllers (no NTLM) via the new ‘rdr’ syntax:

# /usr/likewise/bin/lwio-tool rdr openpipe //<domain_controller>/NETLOGON

The ‘lwio-tool’ usage in OneFS 9.5 is as follows:

# /usr/likewise/bin/lwio-tool -h

Usage: lwio-tool <command> [command-args]

   commands:

    iotest rundown

    rdr [openpipe|openfile] username@password://domain/path

    srvtest transport [query|start|stop]

    testfileapi [create|createnp] <path>

Author: Nick Trimbee

There has been a tremendous surge of information about artificial intelligence (AI), and generative AI (GenAI) has taken center stage as a key use case. Companies are looking to learn more about how to build architectures to successfully run AI infrastructures. In most cases, creating a GenAI solution involves fine-tuning a pretrained foundational model and deploying it as an inference service. Dell recently published a design guide – Generative AI in the Enterprise – Inferencing, that provides an outline of the overall process.

All AI projects should start with understanding the business objectives and key performance indicators. Planning, data prep, and training make up the other phases of the cycle. At the core of the development are the systems that drive these phases – servers, GPUs, storage, and networking infrastructures. Dell is well equipped to deliver everything an enterprise needs to build, develop, and maintain analytic models that serve business needs.

GPUs and accelerators have become common practice within AI infrastructures. They pull in data and training/fine-tune models within the computational capabilities of the GPU. As GPUs have evolved, their ability to handle larger models and parallel development cycles has evolved. This has left a lot of us wondering - how do we build an architecture that will support the model development that my business needs? It helps to understand a few parameters.

Defining business objectives and use cases will help shape your architecture requirements.

• The size and location of the training data set
• Model size in number of parameters and type of model being trained/fine-tuned
• Training parallelism and time to complete the training/fine-tuning.

Answering these questions helps determine how many GPUs are needed to train/fine-tune the model. Consider two main factors in GPU sizing. First is the amount of GPU memory needed to store model parameters and optimizer state. Second is the number of floating-point operations (FLOPs) needed to execute the model. Both generally scale with model size. Large models often exceed the resources of a single GPU and require spreading a single model over multiple GPUs.

Estimating the number of GPUs needed to train/fine-tune the model helps determine the server technologies to choose. When sizing servers, it’s important to balance the right GPU density and interconnect, power consumption, PCI bus technology, external port capacity, memory, and CPU. Dell PowerEdge servers include a variety of options for GPU types and density. PowerEdge XE Servers can host up to 8 NVIDIA H100 GPUs in a single server GenAI on PowerEdge XE9680, as well as the latest technologies, including NVLink, NVIDIA GPUDirect, PCIe 5.0, and NVMe disks. PowerEdge mainstream servers range from two to four GPU configurations, offering a variety of GPUs from different manufacturers. PowerEdge servers provide outstanding performance for all phases of model development. Visit Dell.com for more on PowerEdge Servers.  

Now that we understand how many GPUs are needed and the servers to host them, it’s time to tackle storage. At a minimum, the storage should have capacity to host the training data set, the checkpoints during the model training, and any other data that relates to the pruning/preparing phase. The storage also needs to deliver the data at a rate the GPUs request it. The rate of delivery is multiplied by model parallelism, or the number of models being trained in parallel, and subsequently the number of GPUs requesting the data simultaneously (concurrently). Ideally, every GPU is running at 90% or better to maximize our investment, and a storage system that supports high concurrency is suited for these types of workloads.

Tools such as FIO or its cousin GDSIO (used to understand speeds and feeds of the storage system) are great for gaining hero numbers or theoretical maximums for reads/writes, but they are not representative of performance requirements for the AI development cycles. Data prep and stage shows up on the storage as random R/W, while during the training/fine-tuning phase, the GPUs are concurrently streaming reads from the storage system. Checkpoints throughout training are handled as writes back to the storage. These different points during the AI lifecycle require storage that can successfully handle these workloads at the scale determined by our model calculations and parallel development cycles.

Data scientists at Dell take great effort in understanding how different model development affects server and storage requirements. For example, language models like BERT and GPT have little effect on storage performance and resources, whereas image sequencing and DLRM models have significant or show worst case storage performance and resource demand. For this, the Dell storage teams focus testing and benchmarking on AI Deep Learning workflows based on popular image models like ResNet with real GPUs to understand the performance requirements needed to deliver data to the GPU during model training. The following image shows an architecture designed with Dell PowerEdge servers and networking with PowerScale scale-out storage.

Dell PowerScale scale-out file storage is especially suited for these workloads.  Each node in a PowerScale cluster delivers equivalent performance as the cluster and workloads scale. The following images show how PowerScale performance scales linearly as GPUs are increased, while the performance of each individual GPU remains constant. The scale-out architecture of PowerScale file storage easily supports AI workflows from small to large.

Figure 1.  PowerScale linear performance 

Figure 2.  Consistent GPU performance with scale

The predictability of PowerScale allows us to estimate the storage resources needed for model training and fine-tuning. We can easily scale these architectures based on the model type and size along with the number and type of GPUs required.

Architecting for small and large AI workloads is challenging and takes planning. Understanding performance needs and how the components in the architecture will perform as the AI workload demand scales is critical.

Author: Darren Miller

Now in its 9th generation, Emmy-award-winning Dell PowerScale storage has been field proven in media workflows for over two decades and is the world’s most flexible¹, efficient², and secure³ scale-out NAS solution.  

Our partnership with Marvel Studios is a wonderful example of the innovations we collaborate on with leading media and entertainment companies around the world—with PowerScale as the preeminent storage solution that enables data-driven workflows to accelerate content-creation pipelines. 

Hear about Marvel Studios’ implementation of PowerScale directly in this educational video series from The Advanced Imaging Society: 

The PowerScale OneFS advantage

The underlying OneFS file system leverages the foundations of clustered high-performance computing to solve the challenges of data protection at scale and client accessibility in a massively parallelized way. In practice, a single namespace that easily scales out with nodes to increase performance and capacity is a fundamentally game-changing architecture. 

Media workflows require increased levels of access for the applications and users to provide for workflow collaboration in balance with security that doesn’t impede performance. Further, performance and access can’t be impeded even during hardware failures such as drive rebuilds or system upgrades to ensure that production work can continue uninterrupted while maintenance is being performed in the background.  

Maximizing uptime correlates with fundamental business needs including meeting project timelines and budgets while ensuring that personnel have access to the content at the required performance levels, even during a background maintenance activity. 

As a sufficiently advanced enterprise-class solution, PowerScale incorporates these capabilities to eliminate complexity and provide for increased uptime through its self-healing and self-managing functionality. (For more information, see the PowerScale OneFS Technical Overview.) This takes many of the traditional storage management burdens off the administrator’s plate, lowering the overhead and time needed to maintain storage, which is often increasing in size and scale.

While the benefits of collaboration over Ethernet-based storage are inherent in PowerScale, the user experience is also paramount in correlation with the performance of the network and underlying storage system. Operations such as playout and scrubbing need to perform reliably with no frame drops while providing response times that look and feel equivalent to working from the local workstation. 

As I’ve participated in the development of media storage solutions from SCSI, Fibre Channel, iSCSI, SAN, and Ethernet, I’ve been able to test and compare solutions over the years and have closely watched the evolution and trends of these protocols in relation to their ability to support media workflows. 

In 2007, I demonstrated real-time color grading with uncompressed content over 1 Gb Ethernet, the first of its kind. At the time, using Ethernet-based storage for color grading was largely unheard of, with few applications supporting it. That was more of an exercise to showcase the art of the possible in comparison with Fibre Channel-based solutions. The wider adoption of Ethernet for this particular use case was not yet of high interest because Ethernet speeds still needed to evolve. However, 1 Gb Ethernet was very appropriate for compressed media workflows and rendering, which were well aligned with the high-performance, scale-out design of PowerScale. 

As 10 Gb Ethernet speeds became prevalent, there was a significant uptick in the adoption of Ethernet-based storage compared to Fibre Channel-based solutions for media use cases. I also started to see more datasets being moved over Ethernet rather than by sneaker net, physically delivering drives and tapes between locations. This led to cost and time savings for project timelines and budgets, among other benefits. 

Fast-forward to 2014, when, with the OneFS 7.1.1 version supporting SMB multi-channel, we were able to use two 10 Gb Ethernet connections to support a stream of full resolution uncompressed 4K, whereas a single 10 Gb connection was only capable of supporting 2K full resolution streams. This began an adoption trend of Ethernet solutions for 4K full-resolution workflows. 

In 2017, with the release of the F800 All-Flash PowerScale and OneFS 8.1, 40 Gb Ethernet speeds were supported. The floodgates were unlocked for media workflows. Multiple full-resolution 2K and 4K workflows could run on a single shared OneFS namespace with uncompromised performance. Workload consolidation could be performed and started to eliminate the need for multiple discreet storage solutions that were each supporting different parts of the pipeline, bringing all those together under a single unified OneFS namespace to streamline environments.  

Complete pipeline transformations were taking place and began to replace iSCSI and Fibre Channel-based solutions at an accelerated pace, as those solutions were siloed within workgroups and inflexible with the emerging needs of collaboration. When the PowerScale F900 NVMe solution supporting 100 Gb Ethernet came out in 2021, the technology was set to change the industry yet again.  

With the increasing prevalence of 100 Gb Ethernet over these past few years, performance parity with Fibre Channel-based solutions to support full-resolution 4K, 8K, and all related media workflows in between is no longer in question. Native Ethernet-based solutions are preferred for many reasons—including cloud capability, scale, cost, and supportability—to facilitate unstructured media datasets, leveraging the abundance of network engineering talent in comparison to Fibre Channel-trained engineers.  

With reliability, performance, and shared access for collaboration delivering uncompromised benefits, we now look to several PowerScale storage capabilities that enable rich media ecosystems to be further streamlined and flourish. 

There are four additional key areas of focus and their underlying feature sets that are increasingly important to today’s media ecosystems. They encompass: 

• Security, physical and logical 
• API orchestration 
• Data movement 
• Quality of service 

Security

• In relation to PowerScale being the world’s most secure scale-out NAS solution and in alignment with the Trusted Partner Network (TPN), the OneFS operating system meets or exceeds the Motion Picture Association (MPA) Content Security Best Practices (CSBP) in all relevant areas of the Content Security Model (CSM).⁴
  
  
• Further, I’m seeing greater adoption of self-encrypting drives (SEDs), which provide encryption at the physical layer. Security auditing and multi-factor authentication are among the features being employed to protect the logical layer. Specifically, auditing has been available and used for many years now to provide a range of benefits beyond security.  
  
  
• The real-time audit logs can be parsed to provide performance introspection, analysis, and user-trend insights in addition to identifying abnormal data-access patterns. The logs can also be used to correlate with levels of access to specific projects and files, correlating back to business-level insights and reports as well. 
  
  
• I’m also keen on mentioning the OneFS embedded firewall, which provides connection-level protection at the storage network layer. Firewalls are typically employed in front of the storage or further upstream on the network, so having an additional firewall within the storage that protects the network ports on the storage itself is a powerful layer of security. 

For more information about OneFS security, see Dell PowerScale OneFS: Security Considerations.

Data orchestration

• Data orchestration is paramount to workflow automation. If aspects of workflows can be automated and don’t require an operator to make a decision, they should be automated to remove the possibility for operator error and streamline the environment to accelerate workflows where possible. 
  
  
• Orchestration is enabled through API calls to integrate PowerScale with the environment’s application layers, which can take mundane repeatable tasks off the operator’s plate and increase workflow efficiency. 

For more information about PowerScale data orchestration, see the OneFS documentation on Dell Support.

Data movement

• Data movement is integral to media workflows today, and for that we look to the high-performance, highly reliable SyncIQ protocol embedded in PowerScale OneFS. SyncIQ facilitates secure, parallelized transfer of datasets between PowerScale solutions, providing strong benefits for media workflows. Replication policies can be set up or initiated ad hoc to transfer datasets between PowerScale solutions over the network.  
  
  
• With SyncIQ, PowerScale is both a storage platform and a data transfer engine, so additional servers and transfer applications, which would incur additional cost and management overhead, don’t need to be implemented in front of PowerScale.

The PowerScale Backup and Recovery Guide provides more information about data movement capabilities in OneFS.

Quality of service

• Quality of service is increasingly important and has always been of interest for media workflows. SmartQoS is an embedded OneFS feature that monitors front-end protocol traffic over NFS, SMB, and S3. It allows limits to be set for the number of protocol operations to tie them back to performance SLAs, prioritization of workloads, and support for throttling to prevent specific clients from saturating a connection. I’ve seen an unthrottled copy job use the available connection bandwidth and interrupt a playout, so that’s an example of a use case where SmartQoS can be applied. 
  
  
• Clients can be logically grouped and monitored with all kinds of metrics being captured to quantify and profile workloads. Introspection into read and write latencies, IOPS, and many other metrics on a per-protocol, path, IP address, user, and group basis can be captured and correlated in real time. Metrics can be tracked and enforced to provide quality of service for specific classes of users and workflows, which can all be defined to manage workloads. 

For more information about quality of service in OneFS, see this blog post: OneFS SmartQoS.

Summary

The capabilities of PowerScale storage with OneFS are delivering unparalleled scale and feature benefits that elevate the capabilities of media entertainment use cases from the highest performance workflows to highly dense archives. Standardization on this enterprise-class, secure, and collaborative platform is the key to unlocking innovation and advancing your media pipelines. 

¹ Based on internal analysis of publicly available information sources, February 2023. CLM-0013892.

² Based on Dell analysis comparing efficiency-related features: data reduction, storage capacity, data protection, hardware, space, lifecycle management efficiency, and ENERGY STAR certified configurations, June 2023. CLM-008608.

³ Based on Dell analysis comparing cyber-security software capabilities offered for Dell PowerScale vs. competitive products, September 2022.

⁴ Dell Technologies Executive Summary of Compliance with Media Industry Security Guidelines, https://www.delltechnologies.com/asset/en-ae/products/storage/briefs-summaries/tpn-executive-summary-compliance-statement.pdf.

Author: Brian Cipponeri, Global Solutions Architect 
Dell Technologies – Unstructured Data Solutions 

Welcome to the first in a series of blog posts to reveal some helpful tips and tricks when supporting media production workflows on PowerScale OneFS.

OneFS has an incredible user-drivable toolset underneath the hood that can grant you access to data so valuable to your workflow that you'll wonder how you ever lived without it.

When working on productions in the past I’ve witnessed and had to troubleshoot many issues that arise in different parts of the pipeline. Often these are in the render part of the pipeline, which is what I’m going to focus on in this blog.

Render pipelines are normally fairly straightforward in their make-up, but they require everything to be just right to ensure that you don’t starve a cluster of resource, which, if your cluster is at the center of all of your production operations can cause a whole studio outage, causing impact to your creatives, revenue loss, and unnecessary delays in production.

Did you know that any command that is run on a OneFS cluster is an API call down to the OneFS API. This can be observed if you add the --debug flag to any command that you run on the CLI. As shown here, this displays the call information that was sent to gather the information requested, which is helpful if you're integrating your own administration tools into your pipeline.

# isi --debug statistics client list
        2023-06-22 10:24:41,086 DEBUG rest.py:80: >>>GET ['3', 'statistics', 'summary', 'client']
        2023-06-22 10:24:41,086 DEBUG rest.py:81:    args={'sort': 'operation_rate,in,out,time_avg,node,protocol,class,user.name,local_name,remote_name', 'degraded': 'False', 'timeout': '15'}
        body={}
        2023-06-22 10:24:41,212 DEBUG rest.py:106: <<<(200, {'content-type': 'application/json', 'allow': 'GET, HEAD', 'status': '200 Ok'}, b'n{\n"client" : [  ]\n}\n')

There are so many potential applications for OneFS API calls, from monitoring statistics on the cluster to using your own tools for creating shares, and so on. (We'll go deeper into the API in a future post!)

When we are facing production-stopping activities on a cluster, they're often caused by a rogue process outside the OneFS environment that is as yet unknown to us, which means we have to figure out what that process is and what it is doing.

In walks isi statistics.

By using the isi statistics command, we can very quickly see what is happening on a cluster at any given time. It can give us live reports on which user or connection is causing an issue, how much I/O they're generating as well as what their IP is, what protocol they’re connected using, and so on.

If the cluster is experiencing a sudden slowdown (during a render, for example), we can run a couple of simple statistics commands to show us what the cluster is doing and who's hitting it the hardest. Some examples of these commands are as follows:

isi statistics system --n=all --format=top

Displays all nodes’ real-time statistics in a *NIX “top” style format:

# isi statistics system --n=all --format=top
Node   CPU SMB FTP HTTP NFS HDFS  S3 Total NetIn NetOut DiskIn DiskOut
 All 33.7% 0.0 0.0  0.0 0.0   0.0 0.0   0.0 401.6  215.6     0.0     0.0
   1 33.7% 0.0 0.0  0.0 0.0   0.0 0.0   0.0 401.6  215.6     0.0     0.0

isi statistics client list --totalby=UserName --sort=Ops

This command displays all clients connected and shows their stats, including the UserName they are connected with. It places the users with the highest number of total Ops at the top so that you can track down the user or account that is hitting the storage the hardest.

# isi statistics client --totalby=UserName --sort=Ops
 Ops     In  Out  TimeAvg   Node  Proto  Class   UserName  LocalName  RemoteName
-----------------------------------------------------------------------------
12.8 12.6M 1.1k  95495.8     *       *      *      root          *           *
-----------------------------------------------------------------------------

isi statistics client --UserName=<username> --sort=Ops

This command goes a bit further and breaks down ALL of the Ops by type being requested by that user. If you know the protocol that the user you’re investigating is using we can also add the operator “--proto=<nfs/smb>” to the command too.

# isi statistics client --user-names=root --sort=Ops
 Ops     In   Out  TimeAvg   Node  Proto           Class  UserName        LocalName    RemoteName
----------------------------------------------------------------------------------------------
 5.8   6.1M 487.2 142450.6     1   smb2           write      root 192.168.134.101 192.168.134.1
 2.8 259.2 332.8    497.2      1   smb2      file_state      root 192.168.134.101 192.168.134.1
 2.6 985.6 549.8  10255.1      1   smb2          create      root 192.168.134.101 192.168.134.1
 2.6 275.0 570.6   3357.5      1   smb2  namespace_read      root 192.168.134.101 192.168.134.1
 0.4   85.6  28.0   3911.5      1   smb2 namespace_write      root 192.168.134.101 192.168.134.1
----------------------------------------------------------------------------------------------

The other useful command, particularly when troubleshooting ad hoc performance issues, is isi statistics heat.

isi statistics heat list --totalby=path --sort=Ops | head -12

This command shows the top 10 file paths that are being hit by the largest number of I/O operations.

# isi statistics heat list --totalby=path --sort=Ops | head -12
  Ops   Node  Event  Class Path
----------------------------------------------------------------------------------------------------
141.7     *       *      * /ifs/
127.8     *       *      * /ifs/.ifsvar
 86.3      *      *      * /ifs/.ifsvar/modules
 81.7      *      *      * SYSTEM (0x0)
 33.3      *      *      * /ifs/.ifsvar/modules/tardis
 28.6      *      *      * /ifs/.ifsvar/modules/tardis/gconfig
 28.3      *      *      * /ifs/.ifsvar/upgrade
 13.1      *      *      * /ifs/.ifsvar/upgrade/logs/UpgradeLog-1.db
 11.9      *      *      * /ifs/.ifsvar/modules/tardis/namespaces/healthcheck_schedules.sqlite
 10.5      *      *      * /ifs/.ifsvar/modules/cloud

Once you have all this information, you can now find the user or process (based on IP, UserName, and so on) and figure out what that user is doing and what's causing the render to fail or high I/O generation. In many situations, it will be an asset that is either sitting on a lower-performance tier of the cluster or, if you're using a front side render cache, an asset that is sitting outside of the pre-cached path, so the spindles in the cluster are taking the I/O hit.

For more tips and tricks that can help to save you valuable time, keep checking back. In the meantime, if you have any questions, please feel free to get in touch and I'll do my best to help!

Author: Andy Copeland
Media & Entertainment Solutions Architect

The OneFS key manager is a backend service that orchestrates the storage of sensitive information for PowerScale clusters. To satisfy Dell’s Secure Infrastructure Ready requirements and other public and private sector security mandates, the manager provides the ability to replace, or rekey, cryptographic keys.

The quintessential consumer of OneFS key management is data-at-rest encryption (DARE). Protecting sensitive data stored on the cluster with cryptography ensures that it’s guarded against theft, in the event that drives or nodes are removed from a PowerScale cluster. DARE is a requirement for federal and industry regulations, ensuring data is encrypted when it is stored. OneFS has provided DARE solutions for many years through secure encrypted drives (SEDs) and the OneFS key management system.

A 256-bit key (MK) encrypts the Key Manager Database (KMDB) for SED and cluster domains. In OneFS 9.2 and later, the MK for SEDs can either be stored off-cluster on a KMIP server or locally on a node (the legacy behavior).

However, there are a variety of other consumers of the OneFS key manager, in addition to DARE. These include services and protocols such as: