Dell Technologies is excited to unveil the latest iteration of its object storage solution, Dell ObjectScale 1.4.0. Engineered to cater to the rigorous requirements of enterprise-level storage, Dell ObjectScale 1.4.0 boasts a software-defined and containerized architecture. This advanced design ensures exceptional reliability, performance, and scalability, further solidifying Dell Technologies' dedication to delivering superior storage solutions.

Dell ObjectScale offers diverse deployment options that cater to a plethora of infrastructural needs, providing flexibility and efficiency in management and scaling enterprise object storage.

New features in Dell ObjectScale 1.4.0

Dell ObjectScale 1.4.0 introduces a host of new features and enhancements aimed at improving serviceability, security, and manageability:

Monitoring and reporting enhancements

The enhanced management API offers comprehensive monitoring capabilities, allowing for better visibility into cluster status, node-based inventory details, and component health. Additionally, new audit messages for cluster user set password events and new alerts for successful and failed jobs have been introduced. Users can now access various functions through the ObjectScale portal or a REST client, enhancing serviceability and oversight. This enhancement benefits administrators by enabling proactive maintenance and troubleshooting and ensuring optimal performance.

Dell ObjectScale portal improvements

The portal now provides a complete inventory of all nodes, disks, pods, and PVCs, coupled with enhanced Grafana UI dashboards for in-depth data visualization and performance monitoring. This improvement aids in efficient resource management and planning, allowing administrators to make informed decisions and optimize cluster performance.

S3 bucket deletion

A new API facilitates the deletion of buckets containing data, streamlining the process and offering better monitoring capabilities. Previously, a standard S3 bucket could only be deleted if it was empty. The new feature allows for the deletion of non-empty buckets through a background task with configurable speed and resource consumption. This feature is particularly beneficial for use cases requiring frequent updates or deletions of stored data as it simplifies operations and enhances flexibility.

ObjectScale replication – improved performance

Enhancements to the replication framework deliver improved throughput for both small and large objects. These enhancements mitigate the growth of replication backlogs and optimize resource utilization, thereby ensuring data protection without adversely affecting customer applications.

Upgrade and installation enhancements

The refined deployment and installation process, including a reduced package size and streamlined upgrade process, improves the customer experience. The appliance and software bundle deployments provide a 1-click upgrade feature for the application layer, simplifying upgrades and reducing operational complexity and downtime.

ObjectScale ‘As An Application’ improvements

Version 1.4 provides management IAM support for deployments on Red Hat OpenShift, offering a simplified management user experience and a common IAM model across all ObjectScale deployment models. This enhancement is beneficial for organizations using OpenShift as it provides consistent and simplified management across different deployment environments.

Storage reclamation process improvements

The garbage collection (GC) process reclaims space from discarded full chunks or chunks containing a mixture of deleted and non-deleted object fragments no longer referenced. Enhancements to the GC process and verification scanner improve the efficiency of the storage reclamation process. This benefit is crucial for long-term data retention and storage optimization as it ensures efficient use of storage resources and reduces costs.

Security: cipher security update

ObjectScale has upgraded its cryptographic module to BSAFE Crypto-J 6.3, ensuring FIPS 140-2 compliance for Data-at-Rest Encryption. This enhancement is essential for maintaining stringent security measures, providing robust encryption and enhancing data protection.

Platform latest version support

Dell ObjectScale 1.4 now supports the latest versions of Red Hat OpenShift (4.14) and Kubernetes (v1.27), allowing for platform upgrades to the most current releases. This benefit ensures compatibility with and access to the latest features and improvements, enhancing overall system performance and security.

Conclusion

Dell ObjectScale 1.4.0 offers a sophisticated, enterprise-grade object storage solution that is flexible, scalable, and secure. With its robust architecture and integration with Kubernetes, it provides a compelling option for organizations seeking to optimize their object storage infrastructure.

Author: Cris Banson

IDC predicts that the Global Datasphere will grow to 221 zettabytes by 2026, more than 90% of which is unstructured in nature. ECS, the leading enterprise-class object storage platform from Dell Technologies, has been engineered to support both traditional and next-generation workloads. ECS delivers the capabilities of the public cloud with the command and control of a private cloud infrastructure as an S3-compatible, globally scalable object store.

New Features in ECS 3.8.1

The latest release of ECS 3.8.1 introduces a range of innovative features and enhancements that make significant strides in advancing the field of enterprise object storage solutions.

Azure AD OBO Support

Today more customers are moving to Azure AD, and more apps are using OIDC (OpenID Connect), so that they can talk to a service provider like ECS that supports SAML (Security Assertion Markup Language). Apps in this environment are using an Azure AD On Behalf Of (OBO) workflow to exchange their OIDC token for a SAML assertion. With the support of this new workflow, our customers can integrate their S3 applications to authenticate identity.

The OAuth 2.0 On-Behalf-Of flow (OBO) is ideal for use cases where an application invokes a service/web API and needs to call another service/web API. The idea is to propagate the delegated user identity and permissions through the request chain. For the middle-tier service to make authenticated requests to the downstream service, it needs to secure an access token from the Microsoft identity platform, on behalf of the user.

ECS IAM features S3 work with SAML identity providers to handle authentication and SAML Assertion generation. It provides the following for applications:

• Authenticate with an identity provider and if successful receive a SAML Assertion
• Use the SAML Assertion to make a call to the ECS Secure Token Service (STS) API (AssumeRoleWithSAML Method) to retrieve a temporary set of credentials thus allowing the caller to assume a role
• Perform ECS API calls that the role allows using the temporary credentials. 

Note: In the SAML model there are two main roles for a participant: Identify Provider and Service Provider.  Based on the ECS IAM SAML design ECS acts as the Service Provider and Azure AD acts as the Identity Provider and generates SAML Assertions.

HDFS Deprecated in ECS

Starting from ECS 3.8.1, we will remove the HDFS support, because customers have moved to ECS for Hadoop by S3a. We made the below changes:

• Disabled HDFS head port (9040) on ECS
• Removed CMF (Configuration Framework) support for port enable/disable with HDFS
• Discontinued Datahead service on port 9040
• Removed HDFS client jar file from downloads

Note: When ECS is upgraded to 3.8.1, HDFS will stop working if it is being used.

New CAS Consistency Option

ECS uses strong consistency, and object concurrent conflicts are resolved by redirecting all object operations to the object owner. However, operations might experience additional latency if an object or bucket owner is at a remote VDC. The is a problem for some CAS applications that are very sensitive to read latency.

Starting from ECS 3.8.1, we introduce a new feature which allows customers to set a CAS bucket with a new consistency mode. The greatest benefit to customers is improving read latency when an object or bucket owner is at a remote VDC (virtual Data Center). With the new feature enabled, read performance should improve, because ECS will no longer check the source bucket or object owner that strong consistency typically requires.

Below is the UI configuration page to create a CAS bucket with the CAS consistency mode. The new CAS consistency only supports the CAS buckets which enable ADO RW. The create, update, and delete operations are still redirected to object owner zone.

Simplified Bucket Deletion

The task of deleting a bucket is simplified by incorporating the object deletion process. Customers no longer need to empty a bucket prior to requesting a bucket deletion. Through the user interface, a customer may delete a bucket in ECS, even if it is not empty.

A new UI is introduced about deleting bucket dialog as below. During S3 bucket deleting, 

• Bucket access is set to read only.
• No property changes on bucket are allowed.
• MPUs are aborted.
• Object/versions are removed.
• User permissions, object lock, governance, compliance, retention, and legal hold are honored during delete.
• Once all objects are deleted, the bucket will be removed.
• If objects cannot be deleted, the bucket will be put back into writable state.

A filesystem enabled bucket is also supported in the simplified bucket deletion feature. NFS exports must be removed before deleting. During the deletion process, NFS access will not be allowed (read or write).  

Fabric Improvements with Mixed Memory Cluster

The ECS fabric improvements with mixed memory cluster features resolves the problem where 192 GB nodes get set to a 64 GB profile. This occurred when certain service procedures were run like node expansions, node replacement or software upgrades. With this improvement, service procedures leave the allocated memory profile aligned to what's available physically on the node.

Conclusion

Dell ECS offers a sophisticated solution for deploying and managing enterprise-grade object storage. With its well-designed architecture and robust protective features, it presents a compelling option for organizations in pursuit of flexibility, scalability, performance, and security in their object storage solutions.

Please refer to the ECS 3.8.1 release note for more information about the new features.

This blog is a continuation of the Dell ObjectScale Data Path Overview Part I. Here, I will cover data protection and dataflow. If you haven’t already, feel free to check out the Dell ObjectScale Data Path Overview Part I to get some basic knowledge about chunks, metadata, and the B+ tree.

Data protection methods

An object created within ObjectScale includes writing data and metadata. ObjectScale metadata includes journal chunks and btree chunks. Each is written to a different logical chunk that will contain ~128 MB of data from one or more objects. ObjectScale uses a combination of triple mirroring and erasure coding to protect the data.

• Triple mirroring ensures that three copies of data are written, protecting against two node failures.
• Erasure coding provides enhanced data protection from disk and node failures, using the Reed Solomon erasure coding scheme which breaks up chunks into data and coding fragments that are equally distributed across nodes.
  • ObjectScale uses 12+4 erasure coding.  That means a chunk is broken into 12 data segments and another 4 coding (parity) segments.

Depending on the size and type of data, data is written using one of the data protection methods shown in the following table.

Table 1. Data protection methods based on data type and size

Note: In the ObjectScale appliance (XF960) with NVMe architecture, the object data will have inline erasure coding when >=44MB. In this blog, we will talk about 128MB chunk as default.

Triple mirroring

The triple-mirror write method applies to the ObjectScale journal and Btree chunks, of which ObjectScale creates three replica copies. Each replica copy is written to a single disk on different nodes. This method protects the chunk data against two-node or two-disk failures. 

Triple mirroring plus in-place erasure coding

This write method is applicable to the data from any object that is less than 128 MB in size.

As an object is created, it is written as follows:   

1. One copy is written in fragments that are spread across different nodes and disks.
2. A second replica copy of the chunk is written to a single disk on a node.
3. A third replica copy of the chunk is written to a single disk on a different node.
4. Other objects are written to the same chunk until it contains ~128 MB of data. The erasure coding scheme calculates coding (parity) fragments for the chunk and writes these fragments to different disks.
5. The second and third replica copies are deleted from disk.
6. After this sequence is complete, the chunk is protected by erasure coding.

Figure 1. Process of triple mirroring plus in-place erasure coding 

Inline erasure coding

This write method is applicable to the data from any object that is 128 MB or larger. Objects are broken up into 128 MB chunks. The Reed Solomon erasure coding scheme calculates coding (parity) fragments for each chunk. Each fragment is written to different disks across the nodes.

Any remaining portion of an object that is less than 128 MB is written using the triple mirroring plus in-place erasure coding scheme. As an example, if an object is 150 MB, 128 MB is written using inline erasure coding. The remaining 22 MB is written using triple mirroring plus in-place erasure coding.

Checksums

Checksums are done per write-unit, up to 2 MB. During write operations, the checksum is calculated in memory and then written to disk. On reads, data is read along with the checksum. The checksum is then calculated in memory from the data read and compared with the checksum stored in disk to determine data integrity. Additionally, the storage engine runs a consistency checker periodically in the background and does checksum verification over the entire dataset.

Data flow

ObjectScale was designed as a distributed architecture and includes a built-in load balancer (Metallb by default) that chooses the node in a cluster that will respond to a read or write request.

The following figure and steps provide a high-level overview of a write dataflow.

Figure 2. High-level overview of a write dataflow

1. A write object request is received. In this example, Node 1 processes the request.
2. Depending on the size of the object, the data is written to one or more chunks. Each chunk is protected using advanced data protection schemes such as triple mirroring plus in-place erasure coding and inline erasure coding. Before writing the data to disk, ObjectScale runs a checksum function and stores the result.
   1. In this example, the object size is 150MB and will be divided into 128MB as chunk 1 and 22MB as chunk 2.
   2. For the chunk 1 with 128MB size, it uses the inline erasure coding scheme (12+4).
   3. For the chunk 2 with 22MB size, it uses the triple mirroring plus in-place erasure coding scheme.
3. After the object data is written successfully, the object metadata will be stored. In this example, Node 3 owns the partition of the object table in which this object belongs. As owner, Node 3 writes the object name and chunk ID to this partition of the object table’s journal logs. Journal logs are triple mirrored, so Node 3 sends replica copies to three different nodes in parallel—Node 2, Node 3, and Node 4 in this example.
4. Acknowledgment is sent to the client.
5. In a background process, the memory table is updated.

The following figure and steps provide a high-level overview of a read dataflow.   

Figure 3. High-level overview of a read dataflow

1. A read request is received for ImgA. In this example, Node 1 processes the request.
2. Node 1 requests the chunk information from Node 2 (object table partition owner for ImgA).
3. Knowing that ImgA is in C1 at a particular offset and length, Node 1 requests the chunk’s physical location from Node 3 (chunk table partition owner for C1).
4. Now that Node 1 knows the physical location of ImgA, it requests that data from the node or nodes that contain the data fragment or fragments of that file. In this example, the location is Node 4 Disk 1. Then, Node 4 performs a byte offset read and returns the data to Node 1.
5. Node 1 validates the checksum together with the data payload and returns the data to the requesting client.

Note: In step 4, for NVMe architecture like ObjectScale appliance, each node can directly read data from the other node. This architecture contrasts with a hard-drive architecture in which each node can only read its own data store and then transfer to the request node.

Resources

The following Dell Technologies documentation provides additional information related to this blog. Access to these documents depends on an individual’s login credentials. For access to a document, contact a Dell Technologies representative.

• ObjectScale architecture and overview white papers
• Dell ObjectScale Data Path Overview Part I

Author: Jarvis Zhu

Dell ObjectScale is the next evolution of object storage from Dell Technologies. ObjectScale is scale-out, high-performance containerized object storage built for the toughest applications and workloads, from generative AI to analytics and more. ObjectScale features a containerized architecture built on the principles of microservices to promote resiliency and flexibility. It is a branch of Dell ECS’s codebase and has been re-platformed to utilize the native orchestration capabilities of Kubernetes—scheduling, load-balancing, self-healing, resource optimization, and so on.

This blog will focus solely on the data path of ObjectScale. Please refer to the ObjectScale overview and architecture documentation to learn more about ObjectScale.

Before we introduce the data path, let’s first review chunks, metadata, and B+ tree . If you are already familiar with these concepts, please refer to the Dell ObjectScale Data Path Overview Part II, which will introduce data protection and dataflow.

Chunks

A chunk is a logical container that ObjectScale uses to store all types of data, including object data, custom client provided metadata, and system metadata. Chunks contain 128 MB of data consisting of one or more objects, as shown in the following figure.

Figure 1. Composition of a chunk

An object created within ObjectScale includes writing data and metadata. Metadata encompasses journal chunks and btree chunks. Data chunks have either inline coding or triple mirror plus in-place erasure coding protection depending on the size of the object data, and the btree or journal chunks which store the metadata have triple mirror protection. For more information, please refer to the Dell ObjectScale Data Path Overview Part II.

Metadata

Metadata provides information about one or more aspects of the data, including system metadata like owner, creation date, size, and custom metadata. An example of customer metadata could be client=Dell, event=DellWorld, and ID=123.

Both system metadata and custom metadata are stored in B+ tree and journal chunks.

Data management 

ObjectScale maintains the metadata that tracks data locations and transaction history in logical tables and journals.

The tables hold key-value pairs to store information relating to the objects. A hash function is used to do fast lookups of values associated with a key. These key-value pairs are stored in a B+ tree for fast indexing of data locations. In addition, to further enhance query performance of these logical tables, ObjectScale implements a two-level log-structured merge (LSM) tree. Thus, there are two tree-like structures in which a smaller tree is in memory (memory table) and the main B+ tree resides on disk. A lookup of key-value pairs first queries the memory table, and if the value is not memory, it looks in the main B+ tree on disk.

Transaction history is recorded in journal logs which are written to disks. The journals track the index transactions not yet committed to the B+ tree. After the transaction is logged into a journal, the memory table is updated. After a set period of time or when the table in memory becomes full, the table is merged, sorted, or dumped to the B+ tree on disk. A checkpoint is then recorded in the journal.

Figure 2. Metadata management process in ObjectScale

ObjectScale uses several different tables, which can get quite large. To optimize the performance of table lookups, each table is divided into partitions that are distributed across the nodes. The node where the partition is written then becomes the owner or authority of that partition or section of the table.

One such table is a chunk table, which tracks the physical location of chunk fragments and replica copies on disk. The following table shows a sample of a partition of the chunk table. Each chunk identifies its physical location by listing the disk within the node, the file within the disk, the offset within that file, and the length of the data. Chunk ID C1 is erasure coded, and chunk ID C2 is triple mirrored, as shown in table 1.

Table 1. Sample chunk table partition

Another example is an object table, which is used for object name to chunk mapping. The following table shows an example of a partition of an object table that lists the chunk or chunks and shows where an object resides in the chunk or chunks.

Table 2. Sample object table

A service called Atlas, which runs on all nodes, maintains the mapping of table partition owners. The following table shows an example of a portion of an atlas mapping table.

Table 3. Sample Atlas mapping table

We will cover ObjectScale data protection and workflow in the Dell ObjectScale Data Path Overview Part II. 

Resources

The following Dell Technologies documentation provides additional information related to this blog. Access to these documents depends on an individual’s login credentials. For access to a document, contact a Dell Technologies representative.

• ObjectScale architecture and overview white papers
• Dell ObjectScale Data Path Overview Part II

 Author: Jarvis Zhu

Get ready to embark on a revolutionary storage journey as Dell introduces the latest release of Dell ObjectScale 1.3.0. This cutting-edge object storage solution is redefining the possibilities of enterprise-class storage, showcasing Dell Technologies' unwavering commitment to reliability, high performance, and scalability. With its software-defined and containerized architecture seamlessly integrated within Kubernetes, ObjectScale empowers organizations to adapt swiftly to the ever-evolving business landscape.

New Features in ObjectScale 1.3

The latest release of ObjectScale 1.3 introduces a range of innovative features and enhancements that make significant strides in advancing the field of enterprise object storage solutions.

New deployment option with the powerful ObjectScale Hardware Appliance

ObjectScale can now be deployed on the newly introduced ObjectScale XF960 All Flash Array appliance. The Dell ObjectScale XF960 is an advanced, scale-out object storage solution tailored for enterprise applications and modern workloads like Generative AI and analytics. It enables accelerated innovation with fully managed Kubernetes, robust S3 compatibility supported by the latest generation of PowerEdge's all-flash performance, power/cooling efficiencies, and comprehensive ObjectScale security measures, including consensus-based protection and SED drives.

The XF960 configuration includes the following features:

• A compact 2U form factor
• Dual Intel Processors with 32 cores per processor
• 256GB of memory
• 24 NVMe TLC 30.72TB drives (for the initial release)
• Dual mirrored boot drives for redundancy
• Network connectivity with 100Gb Back-end and 25Gb Front-end speeds.

The XF960 offers hardware alerting features that empower customers or services to address any arising issues proactively or reactively. These alerts are presented on the ObjectScale Portal interface, ensuring prompt action is taken to resolve potential hardware-related concerns.

Object Lock without versioning

Object Lock serves as a protective measure against inadvertent deletions and unintended overwrites. In the S3 protocol, enabling Object Lock requires the presence of versioning. When Object Lock is enabled alongside versioning, it adheres to the standard S3 protocol, applying locking capabilities to individual object versions, each capable of having a distinct lock type.

With the introduction of ObjectScale 1.3, ObjectScale extensions expand the scope of the conventional S3 protocol. This expansion allows for Object Lock to be applied at the bucket level, even in the absence of versioning. Enabling Object Lock without versioning causes the Object Lock functionality to operate at the bucket level rather than on versions of the bucket. Consequently, overwrite and delete operations are effectively prevented.

Efficient handling of large objects

ObjectScale facilitates the seamless uploading of large objects through the multipart upload (MPU) process using the S3 protocol. MPU enables the construction of large objects from smaller parts that can be uploaded at any time and in any order.

With the introduction of ObjectScale 1.3, significant enhancements have been made to improve efficiency, reduce timeouts, and optimize memory usage during MPU processing. Replicated objects can now reach a size of up to 50TiB, with support for up to 50,000 parts.

Bucket Logging capability

Bucket Logging is a beneficial feature that systematically captures all requests directed to a source bucket and archives them in a designated target bucket using a consistent format. This feature can be easily configured either through the intuitive ObjectScale Portal user interface or the powerful S3 API. Moreover, this feature offers support for prefixes on the log objects generated in the target bucket, facilitating clear differentiation between source buckets when multiple buckets are logging to the same target. The logs are meticulously organized in a fixed format within the target bucket, with each object precisely indicating the time of delivery.

ObjectScale replication failure alert and Delete Marker replication

In the event of an object replication failure, an alert is automatically generated, furnishing administrators with a comprehensive error code and a detailed description of the failure to facilitate problem resolution. Moreover, the release encompasses the inclusion of the Delete Marker replication setting in the destination bucket, thereby ensuring consistency between the source and destination buckets while promoting space efficiency through space reclamation.

Security: Optional protection features

To fortify the security of the object environment, ObjectScale 1.3 incorporates a range of optional features:

• Privileged Actions Approval System (PAAS): This workflow-based system mandates approval from a secondary management user before executing high-risk management actions. Its purpose is to prevent any single user from obtaining unrestricted privileges and bypassing security controls.
• Platform Protection Mode: This feature places the ObjectScale system in a lockdown mode, providing robust protection to the platform configuration and infrastructure against alterations by malicious actors. Any predefined configuration action or operating system access requires approval through PAAS. Authorized users can obtain tokens that permit them to open a SSH shell while SSH sessions are prohibited otherwise.
• Account Protection Mode: This mode protects data by enforcing approvals through PAAS for certain predefined data access, account configuration, and bucket configuration changes. It is implemented as a configuration setting on an IAM account and operates on a per-account basis.

ObjectScale Portal user interface enhancements

The latest release brings forth several enhancements to the user interface, encompassing:

• Timeframe Filters now accessible on the ObjectScale and Object Store dashboards
• The ObjectScale Portal application divided into multi-frontend applications, enabling the development of future features seamlessly
• Password change notifications now displayed in days
• Successful login now showcases the user's last login time on the dashboard
• All active user sessions can be viewed and closed, applicable to management users, users from external authorization providers, and approver entity users
• A Performance Monitoring Dashboard exhibiting performance charts for latency, bandwidth, throughput, requests by response codes, and requests by request type methods (Get, Post, Put, Delete, and Head)

Improved storage efficiency

Enhancing space efficiency involves refining the process of space reclamation, thereby reducing the portion of capacity occupied by garage data and its associated metadata. This empowers customers to more accurately estimate their capacity requirements based on actual data needs, fostering a more sophisticated and intellectually driven approach.

SNMP integration

ObjectScale alerts and events can be conveniently accessed via the Simple Network Management Protocol (SNMP) and logged externally to a centralized server, allowing for a comprehensive system management approach. Customers have the flexibility to configure SNMP through the ObjectScale Portal user interface as well as the REST API, enhancing their control over the system.

Conclusion

Dell ObjectScale offers a sophisticated solution for deploying and managing enterprise-grade object storage. With its well-designed architecture, robust protective features, and seamless integration with Kubernetes, it presents a compelling option for organizations in pursuit of flexibility, scalability, performance, and security in their object storage solutions.

Author: Cris Banson

‘A major element fueling the ascent of object storage is the emergence of all-flash object storage and organizations’ growing recognition of the value “fast object” can provide them. All-flash object storage is clearly playing a key role in providing the modern infrastructure necessary to support these new initiatives.‘

This assertion is verified by a report produced by Scott Sinclair of ESG research.

Our latest ObjectScale release features the XF960, an enterprise-class, scale-out, high-performance, all-flash storage system built for the toughest applications and workloads. The XF960 hardware stack includes the server, switch, rack-mount equipment, and appropriate power cables, all optimized to run the ObjectScale software. It is built on Dell Technologies’ PowerEdge R760 server (new 16th generation). 

We’re taking the ObjectScale experience to the next level by providing the latest software innovation as a fully integrated, turnkey solution.

Built with NVMe disks on Dell PowerEdge R760 servers, capacity for the ObjectScale appliance begins at 2.95PB raw capacity and scales up to 11.79PB raw capacity per rack to support the most data-intensive workloads, such as AI, machine learning, IoT, and real-time analytics applications.

ObjectScale appliance configurations include: 

• 2U form factor
• Dual Intel processor with 32 cores per processor
• 256GB memory
• 30TB with 24*NVMe TLC drives 
• Dual mirrored boot drives
• Network: 
  • 100Gb Back-end switches (Dell provided)
  • 25Gb Front-end switches (customer-provided and supported)

Note: The ObjectScale appliance supports a minimum of 4 node deployment and max 16 nodes in a single rack cluster.

Two major performance enhancements with ObjectScale 1.3 and XF960 enable application development teams and data science teams to innovate even faster:

• 16G PowerEdge server performance with full-stack NVMe connectivity. The XF960 embraces the NVMe over TCP protocol for its blazing fast 100Gb backend network, accelerating node-to-node communication and unlocking the true potential of the all-flash system’s throughput rate, especially in large-scale deployments.
• We introduce shared memory in the NVMe IO path of the datahead service in the ObjectScale appliance running software version 1.3 to increase S3 throughput performance.

The ObjectScale appliance offers self-encrypting drives (SEDs), NVMe SSDs that transparently encrypt all on-disk data using an internal key and a drive access password. This encryption protects the system from data theft when a drive is removed. ObjectScale now only supports Secure Enterprise Key Manager (SKEM) in the iDRAC.

The ObjectScale appliance supports encryption for:

• Boot Drive
• Data Drive (providing better hardware deployment performance compared to software based D@RE)
• Software based D@RE (enabled at a system or bucket level)

The ObjectScale appliance supports hardware monitoring by leveraging the iDRAC from servers. It can be enabled and disabled from the UI or API directly. The alerts include a detailed description of the issue with Severity type, Reason, and Impacted resource.

The following workflow summarizes how the hardware monitoring works:

• UI or REST clients issue calls to manage hardware alerts.
• ObjectScale enables hardware alerts.
• iDRAC publishes alerts to the event listener and K8s events.
• The event is finally delivered to SNMP and Dell Support.
• ObjectScale pulls alerts every 60 seconds that display in the UI.

The ObjectScale appliance is deployed with the latest ObjectScale 1.3 software bundle package. For more information about Dell ObjectScale 1.3, see the latest Dell ObjectScale: Overview and Architecture white paper.

Conclusion

The Dell ObjectScale appliance gives organizations flexibility in deploying and managing enterprise-grade object storage. ObjectScale is a distributed system that has a layered architecture: every function is built as an independent layer, which provides horizontal scalability across all nodes and enables high availability. The ObjectScale software provides optimum protection, geo-replication, greater availability, and secure data access. 

Resources

The following Dell Technologies documentation provides additional information about Dell ObjectScale. Access to these documents depends on an individual’s login credentials. For access to a document, contact a Dell Technologies representative.

• ObjectScale and ECS white papers:
  • ObjectScale and ECS Info Hub
• Dell Support resources:
  • ObjectScale 1.3.x Release Notes
  • Dell ObjectScale 1.3.x Administration Guide
  • Dell ObjectScale 1.3.x Security Configuration Guide
  • Dell ObjectScale 1.3.x Rest API Reference
  • Dell ObjectScale XF960 Hardware Guide

Author: Jarvis Zhu

ObjectScale is the evolution of the Dell ECS enterprise object storage architecture, built on containers and refactored to run inside a Kubernetes-native package. The latest generation of object storage software is lighter, faster, and deployable. You can deploy ObjectScale on your Kubernetes (Red Hat OpenShift Container Platform) or SUSE Linux Enterprise Server (SLES) infrastructure. ObjectScale supports the storage, manipulation, and analysis of unstructured data on a massive scale.

To ensure that your SLES infrastructure meets ObjectScale installation requirements, you should run the ObjectScale Qualification Tool as a best practice. The Qualification Tool and related instructions (Readme file for ObjectScale Qualification Tool 1.2.x) are available under Drivers & Downloads on the ObjectScale Software Bundle page on Dell Support.

The preconditions for installing ObjectScale with a SLES infrastructure are as follows:

Cluster hardware requirements

• A minimum of four physical CPU cores; 20 physical cores are recommended.
• A minimum of four nodes with 192 GB RAM each. ObjectScale and the initial object store require at least 400 GB RAM in total. Each additional object store requires another 350 GB RAM.
• 1 x 960 GB (or larger) unused SSD per node.
• A minimum of five unused disks per node for a single object store, preferably the same size disk.

Cluster operating system requirements

• SLES version 15 SP4
• Additional software packages that are not included in the SLES15sp4:
  • Podman
  • Skopeo
  • Jq
  • Haproxy
  • Keepalived
  • Hosa
  • Dcism
• Disabled swap
• The 960 GB or larger disk set in a single logical volume group:
  • /opt requires at least 250 GB.
  • /tmp requires at least 60 GB.
  • /var requires at least 200 GB.
  • Bare-metal CSI storage class PVCs require at least 450 GB.

Other installation considerations

• A pair of bonded 25 Gb network interface cards is recommended for the networking.
• ObjectScale supports both Layer 3 and Layer 2 network configuration. The networking configuration can be set in the yaml file during the ObjectScale installation.
  • ObjectScale only supports Border Gateway Protocol (BGP) for Layer 3 networking.
  • Metallb is used as Layer 2 networking in ObjectScale.
• Network Time Protocol (NTP) accessibility is essential for ObjectScale to operate correctly. Precise time is necessary for consistent clock synchronization between nodes in ObjectScale, and it ensures clean log and journal entries for chunk timestamp values. Errors will result during tenant creation when the system time skew between each node is more than 30 seconds.
• It is important to ensure that the drives are completely empty before an installation. The drives cannot have file systems on them. When the data disks are not cleaned, a PVC pending problem might result.
  • Clean the LV, VG, and PV first in the nodes if the system is reinstalled.
  • Clean the data drives by running the wipefs and dd command.

Note: Do not run the clean command on the drive containing the boot/system partitions.

Resources

For more information, see the following resources:

• ObjectScale: Overview and Architecture white paper
• ObjectScale Software Bundle support Information

Author: Jarvis Zhu

Dell ObjectScale is an object storage solution that represents the evolution of Dell Technologies' commitment to delivering enterprise-class, reliable, and high-performance storage. With its software-defined and containerized architecture, ObjectScale operates seamlessly within Kubernetes, empowering organizations to become more agile and responsive to ever-changing business requirements.

ObjectScale adheres to a set of core design principles that define its exceptional capabilities:

• Global namespace with eventual consistency
• Scale-out capabilities
• Secure multitenancy
• Optimum performance for small, medium, and large objects

In this blog post, we delve into the exciting new features introduced in the Dell ObjectScale 1.2.0 release.

Feature enhancements

This new code release introduces a range of feature enhancements, including: 

• Simpler deployment options and enhanced user experience: The latest release of ObjectScale brings forth a range of improvements to simplify the deployment process and enhance user experience. Key updates include:
  • New deployment option: A faster deployment option, known as the “Software Bundle,” enables the installation of ObjectScale and Kubernetes on a cluster of nodes with SUSE Linux Enterprise Server 15 SP4. This option streamlines the setup process and accelerates the journey toward efficient object storage.
  • Support for Red Hat OpenShift Container Platform: ObjectScale continues to offer the "As an Application" deployment option, providing seamless integration with Red Hat OpenShift Container Platform 4.12. This compatibility ensures that users can leverage the power of ObjectScale within their existing OpenShift environments.

• Streamlined user interface: The ObjectScale Portal boasts a redesigned user interface, offering an intuitive and streamlined workflow for bucket provisioning and management. This enhancement simplifies object storage administration, making it more accessible to users.

• Enhanced performance and efficiency: ObjectScale 1.2.0 introduces several notable performance improvements to optimize efficiency across various use cases:
  • Increased write performance: Users can now enjoy write performance of up to 2 GB/s per node, which enables faster data ingestion and storage.
  • Support for larger objects: ObjectScale now supports objects as large as 30 TB, including replication of these huge objects. This six-fold increase in the maximum object size surpasses the limits set by Amazon S3, making ObjectScale an ideal choice for large-scale projects such as HPC, analytics, AI, and their associated backup requirements.
  • Flexible erasure coding: The latest release includes more flexible erasure coding options, including support for a minimum of four nodes with a 12+4 EC policy. This enables greater data protection and fault tolerance.
  • Logical capacity usage reporting: ObjectScale now provides reporting of logical capacity usage for Dell APEX, facilitating better resource management and planning.
    
    
• Expanded Zero Trust features: ObjectScale 1.2.0 strengthens ObjectScale security and access control capabilities. Highlights include:
  • Active Directory and LDAP support: System users can now benefit from Active Directory and LDAP support, which enables seamless integration with existing user management systems. This simplifies user administration and enhances security.
  • Comprehensive IAM model: ObjectScale introduces an improved Identity and Access Management (IAM) model, accommodating local or management users, external authentication providers, configurable login workflows, user persona roles, password policies, and account lockout configurations. These advancements offer enhanced flexibility and control over access management.
    
    
• Improved reliability and supportability: The latest ObjectScale release focuses on enhancing reliability and supportability:
  • Enhanced maintenance mode handling: ObjectScale now allows up to two nodes to be simultaneously placed in maintenance mode, simplifying maintenance operations and minimizing downtime.
  • Remote Connect Assist: The integration of Pluggable Authentication Module (PAM) enables Remote Connect Assist, empowering remote support technicians to securely access the system using remote secure credentials tokens. This facilitates efficient troubleshooting and support.

Conclusion

Dell ObjectScale 1.2.0 introduces improved deployment options and feature enhancements, solidifying its position as a leading software-defined, containerized object storage platform. For more in-depth information, we encourage you to explore the Dell ObjectScale: Overview and Architecture white paper and the ObjectScale and ECS Info Hub. Embrace these advancements and unlock the full potential of Dell ObjectScale for your organization's evolving storage needs.

Author: Cris Banson

As unstructured data continues to grow exponentially, organizations face challenges of managing and analyzing data in object storage. At some point, it might become necessary to move the data to another ECS cluster or a public cloud to manage object data flexibly and efficiently.

Data movement, also called copy-to-cloud, is a new feature in ECS 3.8.0.1 where a user can copy local object data to an external S3 target, such as a nonfederated ECS or a public hyperscaler. (Currently, only AWS targets are supported.)

The data movement feature allows copying object data to an external S3 target. It is built on the ECS Sync open-source tool, which provides the capability to copy data in parallel. The data movement feature only supports IAM accounts and IAM buckets. Figure 1 shows the data movement solution architecture.

Figure 1.  Data movement architecture 

Data movement is configured as a bucket option in the UI, as shown in figure 2. It can be monitored by an account admin or system admin within the UI. The admin can define policies about source and target buckets and criteria for objects. The admin can also monitor the logs for all copy operations at the object level, including the copy time, source object key, object size, target endpoint, duration, and result of the copy operation (success/failure, error message). There are also alerts that show a summary of all copy operations and errors on any failures.

Note: The target bucket must exist at ECS bucket creation time.

Figure 2.  Config data movement in a bucket

The data movement service can only run in Gen2 or later systems that have been upgraded to 192 GB memory. Data movement policies cannot sync deletes. This means that if an object is deleted from the source bucket, it will not be deleted from the target bucket. The data movement policy runs once an hour; if there are many versions within the hour, only the latest one is copied.  

Dell is extending the ecosystem to support a multicloud experience for Snowflake, which runs in AWS. Dell and Snowflake customers can use on-premises data stored on Dell ECS while keeping their data local or seamlessly copying it to public clouds to leverage Snowflake’s ecosystem of cloud-based data analysis services.

The following workflow shows how Snowflake works with ECS data movement:

Figure 3.  Data movement with Snowflake

1. An application writes data to an ECS local bucket.
2. A data movement policy in ECS is configured to copy all data or a subset of the data to a customer-owned predefined staging bucket within AWS. 
3. Data is written to the staging bucket.
4. This AWS bucket will have S3 notifications configured to notify AWS SQS queue, to which Snowflake is subscribed.
5. A Snowflake data pipeline process called Snowpipe wakes up and ingests the data into Snowflake.
6. Data can then be deleted according to life cycle policy in AWS.
   
   

Conclusion

Whatever the use case for data transfer may be, getting it done fast, reliably, securely, and consistently is important. And no matter how much data you have to move, where it’s located, or how much bandwidth you have, there is an option that can work for you. For a more in-depth look, check out the documentation:

• Dell ECS: Overview and Architecture
• Dell ECS: Data Movement (Copy to Cloud)
• Drive analytics with Snowflake Data Cloud and Dell Technologies storage
• Simplify Data Access and Accelerate Insights with Dell ECS and Snowflake

ECS 3.8.0.1 includes these new and updated features:

• Increased data mobility and flexibility with new data mobility feature
• Expanded external key management (KMIP) support
• Object Lock enhancements for ADO
• Security Token Service (STS) GetFederationToken support
• New hardware disk sizing option for EX500
• Memory upgrade expansion 

Data mobility

Data mobility, also known as copy-to-cloud, is a new feature in version 3.8.0.1. With data mobility, a user can copy local object data to an external S3 target, such as a secondary ECS that is not federated or a public cloud target (currently AWS targets only).

Data mobility is configured as a bucket option in the UI, as shown in Figure 1. It can be monitored by an account admin or system admin within the UI. The admin can define policies about source and target buckets and criteria for objects. The admin can also monitor the logs for all copy operations at the object level, including the copy time, source object key, object size, target endpoint, duration, and result of the copy operation.

Figure 1    Data mobility configuration

We have also extended our ecosystem to support a multi-cloud experience for Snowflake, which runs on the AWS public cloud platform. Dell and Snowflake customers can use on-premises data stored on Dell ECS while keeping their data local or seamlessly copying it to public clouds and use Snowflake’s ecosystem of cloud-based data analysis services.

Expanded external key management support

The Key Management Interoperability Protocol (KMIP) is an extensible communication protocol that defines message formats for the manipulation of cryptographic keys on a key management server.

A new external key management cluster type to support Thales CipherTrust has been added since Gemalto SafeNet KeySecure will end of life on December 31, 2023. ECS customers who are using KeySecure can migrate to CipherTrust Manager.

Object Lock support with ADO enabled

ECS Object Lock enhancements for Access During Outage (ADO) have been added in ECS 3.8.0.1. Object Lock now supports ADO Read Only (RO) by default. For Read Write (RW) mode, ECS continues to deny setting Object Lock on ADO buckets by default. There are flags at the namespace and individual bucket level that require users to agree that they understand the risk of losing locked versions during TSO but would still like to enable this feature. Customers need to refer to the latest ECS Data Access Guide or ask the Dell support team to help enable Object Lock RW in ADO. Once flags are set on a bucket, they cannot be disabled.

The following table shows the Object Lock support matrix:

STS GetFederationToken support

The GetFederationToken API is now part of Security Token Service (STS), along with AssumeRole and AssumeRoleWithSAML. It is called by the IAM user, and it returns a set of temporary security credentials (consisting of an access key ID, a secret access key, and a security token) for that user. This operation federates the user. A typical use is in a proxy application that gets temporary security credentials on behalf of distributed applications inside a corporate network. The ECS 3.8 Administration Guide on Dell Technologies Support provides more details.

New hardware disk sizing option

ECS version 3.8.0.1 extends disk sizing support. Customers can now select an ECS EX500 20 TB disk option.

Memory upgrade expansion

ECS version 3.8.0.1 supports memory upgrade expansion to 192 GB on EX300, EX500, EX3000, and Gen 2 platforms, with the support of Dell Professional Services. For more information, contact your ECS Customer Service representative.

Conclusion

ECS version 3.8.0.1 introduces new and updated platform features. It reiterates the core value proposition of ECS and our broader object portfolio, amplifying the benefits of the new capabilities in the 3.8.0.1 payload and related hardware updates.

For more information, see the ECS: Overview and Architecture White Paper on the ObjectScale and ECS Info Hub.

Dell Technologies is excited to announce the availability of ECS 3.7, released on February 10, 2022. This release introduces the following new and incremental features to the ECS platform:

• S3 select 
• Tokenization support for metadata search
• New hardware disks sizing options

These features continue to build upon the capabilities of the ECS platform and our broader object portfolio. Read on to learn more.

S3 select

One of the key features included with ECS 3.7 is S3 select. This feature enables applications to retrieve only a subset of data from an object by using simple SQL expressions. When you use S3 select to retrieve only the data needed by your application, you can achieve drastic performance increases and network bandwidth saving.

Let’s look at an example scenario with a 2 GB csv object. Without S3 select, your application would have to download the entire 2 GB object and then do the processing on that data. With s3 select, your application issues the SQL select command and gets only a small subset of that data.

You can use S3 select for objects in csv, json and parquet formats. It supports querying gzip/bzip2 compressed objects of the above 3 file types.

S3 select is commonly used by query engine, like presto. A connector like presto can determine if a particular query can be sent directly to the storage. For example: s3 select pushdown.

For AWS S3 compliant partial reads of an object that offloads, these reads query and sort to ECS rather than using compute resources. This feature may provide a performance benefit for use cases where network bandwidth or compute resources are a bottleneck.

S3 select is disabled by default on system with 64 GB memory and enabled by default on systems with 192 GB memory (EXF 900 only). 192 GB memory is a recommendation, however, to enable it on 64 GB memory, you must open a service request (SR) with Dell. We will notify you that memory usage will increase and may impact other operations, and then will enable the feature.

Tokenization support for metadata search

Metadata search with tokenization enables you to search for objects that have a specific metadata value in an array of metadata values. You must choose this method when you create the bucket, and you can include the option when creating the bucket through the S3 create bucket API by including the header x-emc-metadata-search-tokens: true in the request.

For example, in the method without tokenization, with the command x-amzmeta-countries= [france,uk], the value france,uk is considered as a single value.

In the method with tokenization, with the command x-amzmeta-countries= [france,uk], the value france,uk is considered as two values separated with the delimiter.

New hardware disk-sizing options

Finally, ECS 3.7 introduces two new disk-sizing options, each with their own advantages:

• EX500 2 TB and 4 TB disk configurations give you options.
• EXF900 16 TB NVMe disk configuration provides more density, performance and agility.

In closing

The 3.7 release introduces several new and incremental features to the ECS platform, and reiterates the advantages of ECS and our broader object portfolio. 

For more information about the ECS, see the document ECS: Overview and Architecture and the ObjectScale and ECS Info Hub.

Author: Jarvis Zhu

Ransomware is defined as a form of malware that encrypts a victim’s files. The attacker will then demand a ransom from the victim and will only restore access after a payment has been made. These attackers are unscrupulous, always looking for opportunities to exploit weaknesses in potential victim’s defences. 

In the VMware 2020 Cybersecurity Outlook Report, defence evasion is a key tool for these attackers.  So having the right protection in place is paramount. Here are some highlights:

 A wiper attack involves wiping/overwriting/removing data from the victim. Unlike typical cyber-attacks that tend to be for monetary gain, wiper attacks are destructive in nature and often do not involve a ransom. Wiper malware can however be used to cover the tracks of a separate data theft.

Object storage can be regarded as a potential weak point in an organization’s armour. There are some key considerations about object storage that you need to be aware of when putting a security plan in place:

• Object storage platforms typically have no security monitoring tools to make you aware that your data is under threat. 
• Ransomware attackers usually target weak links within IT security and if they somehow obtain the secret key, they can gain access to petabytes of data with no security tools actively monitoring for these potential intrusions.
• Object is often used as a backup and that can make it a soft target because it’s not actively monitored.
• Also, Object can also be used for compliance data for legal hold, making it a target.
• With no notion of native, namespace level snapshots on object platforms recovery is made difficult.
• A few lines of python code are sufficient to attack object storage over S3.
• If your data is important you need to get monitoring in place before your data is attacked unknowingly. 

So how can I ensure my Object storage is safe and actively monitored? 

Protecting against any security threats including ransomware is a layered approach. Currently in Dell EMC Elastic Cloud Storage (ECS) you can use versioning to retain multiple copies of an object to protect against potential attacks.  For many years ECS has provided SEC17a-4(f) level compliance as a WORM-enabled capability when leveraging Amazon S3 API retention extensions.  This WORM capability has been expanded in ECS version 3.6.2 with the addition of S3 Object Lock.  Having these in place will give you a good foundation of protection for your object storage. 

Building on this to offer superior protection to our customers, we have partnered with Superna. With ECS and the Ransomware Defender from Superna, we can monitor user behaviour and detect potential threats to systems quickly. If potential threats do materialize, you can be alerted quickly to disable the user keys to mitigate the threat. Alternatively, you can configure Ransomware Defender to automatically lock the corresponding application user when it detects malicious activity. This can help expedite the recovery process by providing the user with a list of infected objects. The following figure shows a thorough workflow of how Superna can help secure your storage. 

For a demo of the functionality of this partnership, see Eyeglass Ransomware Defender for ECS Overview.

With this partnership we believe we can offer better protection for our customers and allow them to defend themselves against potential external security threats. 

Author: Finbarr O'Riordan  @finbarrorcork on Twitter

The data storage industry has three distinct pillars: block, file, and object. Traditionally, object storage has been leveraged for backup, long term archive, and data lake storage. It has been regarded as “cheap and deep”, but that perception is beginning to change as more and more application workloads suited to object storage are evolving. 

With the introduction of flash-based storage systems, object is now capable of handling high performance workloads at low latencies. Object use cases are expanding into other workloads such as Artificial Intelligence (AI), Deep Learning (DL), Machine Learning (ML), and DevOps. These analytical applications produce a wide range of data types and sizes. With object storage supporting unlimited fully customizable metadata, these features make it easier to locate data for use in AI and ML algorithms, offering enhanced insight for our customers.  

Exponential data growth in recent years presents economic challenges to companies. The scalable architecture of object storage means it’s well suited to handle data growth, allowing customers to add additional nodes when required. 

Elastic Cloud Storage (ECS), Dell EMC’s object storage system, is capable of scaling linearly based on capacity and performance. With flash-based storage, object is a viable alternative for addressing different business needs. Additionally, with the cost of flash storage falling and price points coming close to those of HDDs, the cost difference between the two technologies is narrowing. Customers can leverage higher performance and cost-effective object storage to gain the necessary insight from their data by using AI and ML to extract the intelligence they require to make strategic business decisions.

Over time, the S3 API has become the defacto standard for object storage. S3 allows for sharing of code, software, and tools -- which fits perfectly into the agile development world of DevOps. Having robust APIs is critical, because it allows customers to leverage automation and orchestration methods to respond quickly as required. Object storage APIs that are application and cloud centric allow you to deploy applications rapidly through code and processes. In certain industry verticals such as medical, financial, media, or automotive, flash object can be leveraged for research, financial modeling, streaming services, or CAD use cases. These are ideal for the type of applications that use massive data sets and high performing environments. 

In November 2020, we launched our flash object appliance: the Dell EMC EXF900. It is built with NVMe-based SSDs on Dell EMC PowerEdge servers, and leverages the NVMeOF (non-volatile memory express over fabrics) protocol for its backend network. This enables node-to-node communication and unlocks the true potential of the all-flash system’s throughput rate, especially in large scale deployments. 

With that, we are seeing incredible performance improvements in terms of bandwidth and transactions per second (TPS) when comparing the EXF900 to previous appliance releases. 

We believe the EXF900, with its enterprise performance levels, is perfectly suited to serve the changing workload needs of our customers. The modern data center is evolving and so are the workloads our customers are running on our systems. With its high performance and ability to automate easily, the EXF900 can deliver the scalability and agility they need. 

This significant combination of performance and security features ensures that ECS can support customers’ data-intensive workload requirements while protecting data from malicious activities.

Authors:

Finbarr O’Riordan  @finbarrorcork on Twitter

Christoffer Braendstrup

Dell EMC ECS 3.6.2, available for download since August 5, 2021, includes Object Lock support for our customers. This has been a popular ask and we are delighted to be able to deliver this to our Object Storage install base as it enables them to satisfy many use cases and help them in their daily roles. 

ECS allows you to store objects using a write-once-read-many (WORM) model through Object Lock. This feature prevents objects from being deleted or overwritten for a specified time or indefinitely. Also, Object Lock helps to meet WORM storage related regulatory requirements and adds a protection layer against object modifications and deletion.

ECS Object Lock allows you to manage object retention through retention periods and legal holds. With a retention period, you can specify a period during which an object remains locked. During the specified period, the object is WORM-protected, that is, the object cannot be overwritten or deleted. Legal hold provides the same protection as retention period but is independent from retention period, and does not have an expiration date. Legal hold is retained in objects until you explicitly remove it. Any user who has the appropriate Object Lock permissions can specify retention period and legal hold in objects. 

So, let’s look at a practical example for how we would use these. We may have a situation in a medical environment whereby patient files are not set up correctly for retention purposes, and we have a regulatory requirement to retain these files. To comply with government regulations, we can use the following command to put a legal hold on a bucket that contains the medical records.

S3curl.pl --id=ecsflex -- http://${s3ip} /my-bucket/obj?legalhold -X PUT –d “<LegalHold><Status>on/Status></LegalHold>”

After he places a legal hold on the necessary buckets, our trusty storage administrator should be prepared if an audit is held. 

Next let’s review how we use retention; Object lock has two retention modes:

1. Compliance -- This is regarded as the stricter of the two modes and is primarily targeted for regulatory compliance for certain customer use cases. Users cannot overwrite or delete an object version. Additionally, users can neither remove nor shorten an object retention. However, with s3: PutObjectRetention permission, you can increase an object’s retention period.   
2. Governance -- The governance protection mode is focused on protecting against potential security vulnerabilities such as rogue actors, accidental deletion, or comprised credentials. Ordinary users cannot overwrite or delete an object version, but users with the special privilege of s3:BypassGovernanceRetention can remove or shorten an object retention and delete locked objects. This is, in essence, a superuser privilege, so it is not granted lightly. Additionally, a user with the s3:PutObjectRetention permission can increase the object retention period. 

So, let’s look at a practical example for how we would use these modes. Let’s say from a governance perspective that we have an application owner who is working on an IT skunkworks type project that bore fruit, and they want to make sure that their work is protected and guards against any potential ransomware attack or through accidental deletion. To extend a retention time out to the year 2030 on an existing bucket, they can use this curl command.

S3curl.pl --id=ecsflex -- http://${s3ip} /my-bucket/obj?retention -X PUT –d “<Retention><Mode>GOVERNANCE</Mode><RetainUntilDate>2030-01-01T00:00:00.000Z</RetainUntilDate></Retention>”

This will ensure that the bucket is more secure and protects the user’s work from being overwritten.

ECS Object Lock fulfils some key requirements:

• Enables the management and enforcement of retention policies and legal holds for objects and buckets
• Supports a Governance and a Compliance version of enforcement
• Maintains data integrity and version consistency in multiple sites

We have delivered an API that enables customers to easily manage their Buckets and Objects while protecting themselves and complying to best practice standards. For more detail and other examples, please see our 3.6.2 Dell EMC ECS Data Access Guide.

Notes:

• The ECS Object Lock feature supports only the versioning enabled buckets. 
• There is no ECS user interface for Object Lock. It can be accessed through ECS Object Lock APIs. (In the 3.6.2 Dell EMC ECS Data Access Guide, for the Object Lock API examples, see the section “Object Lock API Examples”; for the list of supported S3 APIs, see the section “S3 API supported and unsupported features”.)
• The locked objects are protected from life cycle deletions.

Author: Finbarr O’Riordan  @finbarrorcork on Twitter

Dell EMC ECS supported WORM (write once, read many) based retention, starting with ECS 2.X. To provide more compatibility with more applications, ECS now supports the object lock feature (starting with ECS 3.6.2), which is compatible with the capabilities of Amazon S3 object lock.

Object lock is designed to meet compliance requirements such as SEC 17a4(f), FINRA Rule 4511(c), and CFTC Rule 17.

Object lock overview

Object lock prevents object version deletion during a user-defined retention period. Immutable S3 objects are protected using object- or bucket-level configuration of WORM and retention attributes. The retention policy is defined using the S3 API or bucket-level defaults. Objects are locked for the duration of the retention period, and legal hold scenarios are also supported.

There are two lock types for object lock:

• Retention period -- Specifies a fixed period of time during which an object version remains locked. During this period, your object version is WORM-protected and can't be overwritten or deleted.
• Legal hold -- Provides the same protection as a retention period, but it has no expiration date. Instead, a legal hold remains in place until you explicitly remove it. Legal holds are independent from retention periods.

There are two modes for the retention period:

• Governance mode -- users can't overwrite or delete an object version or alter its lock settings unless they have special permissions. With governance mode, you protect objects against being deleted by most users, but you can still grant some users permission to alter the retention settings or delete the object if necessary. You can also use governance mode to test retention-period settings before creating a compliance-mode retention period.
• Compliance mode -- a protected object version can't be overwritten or deleted by any user, including the root user in your account. When an object is locked in compliance mode, its retention mode can't be changed, and its retention period can't be shortened. Compliance mode helps ensure that an object version can't be overwritten or deleted for the duration of the retention period.

Object lock and lifecycle

Objects under lock are protected from lifecycle deletions.

Lifecycle logic is made difficult because of the variety of behavior of different locks. From a lifecycle point of view there are locks without a date, locks with date that can be extended, and locks with date that can be decreased.

• For compliance mode, the retain until date can't be decreased, but can be increased.
• For governance mode, the lock date can increase, decrease, or be removed.
• For legal hold, the lock is indefinite.

Some key points for the S3 object lock with ECS

• Object lock requires FS (File System) disabled on bucket in ECS version 3.6.2.
• Object lock requires ADO (Access During Outage) disabled on bucket in ECS version 3.6.2.
• Object lock is only supported by S3 API, not UI workflows in ECS version 3.6.2.
• Object lock only works with IAM, not legacy accounts.
• Object lock works only in versioned buckets.
• Enabling locking on the bucket automatically makes it versioned.
• Once bucket locking is enabled, it is not possible to disable object lock or suspend versioning for the bucket.
• A bucket has a default configuration including a retention mode (governance or compliance) and a retention period (which is days or years).
• Object locks apply to individual object versions only.
• Different versions of a single object can have different retention modes and periods.
• A lock prevents an object from being deleted or overwritten. Overwritten does not mean that new versions can't be created (new versions can be created with their own lock settings).
• An object can still be deleted. It creates a delete marker and the version still exists and is locked.
• Compliance mode is stricter, locks can't be removed, decreased, or downgraded to governance mode.
• Governance mode is less strict, it can be removed, bypassed, or elevated to compliance mode.
• An object can still be deleted, but the version still exists and is locked.
• Updating an object version's metadata, which occurs when you place or alter an object lock, doesn't overwrite the object version or reset its Last-Modified timestamp.
• Retention period can be placed on an object explicitly, or implicitly through a bucket default setting.
• Placing a default retention setting on a bucket doesn't place any retention settings on objects that already exist in the bucket.
• Changing a bucket's default retention period doesn't change the existing retention period for any objects in that bucket.
• Object lock and traditional bucket/object ECS retention can co-exist.

ECS object lock condition keys

Access control using IAM policies is an important part of the object lock functionality. The s3:BypassGovernanceRetention permission is important because it is required to delete a WORM-protected object in Governance mode.  IAM policy conditions have been defined below to allow you to limit what retention period and legal hold can be specified in objects.

ECS object lock API examples

This section lists s3curl examples of object Lock APIs. Put and Get object lock APIs can be used with and without the versionId parameter. If no versionId parameter is used, then the action applies to the latest version.

Conclusion

Dell EMC ECS object lock helps to protect object versions from accidental or malicious deletion, such as a ransomware attack. It does this by allowing object versions to enter a WORM state where access is restricted based on attributes set on the object version.