Dell Technologies has engineered software-defined, enterprise-class, high performance, containerized architecture object storage in the VMware Tanzu platform called ObjectScale.

Dell ObjectScale

VMware and Dell Technologies have worked together to provide a framework for cloud-native object storage in vSphere, using the vSAN Data Persistence platform: Dell ObjectScale on VMware Cloud Foundation with Tanzu. Dell ObjectScale provides native Kubernetes, S3-compatible storage utilizing vSAN. ObjectScale integrates with vSphere, simplifying management for administrators. With rich S3 compatibility and self-service APIs, you can quickly spin up object storage containers to fuel applications, ranging from big data and analytics to ephemeral dev or test sandboxes.

Dell ObjectScale allows organizations to deliver scalable cloud services with the reliability and control of private cloud infrastructure like VMware. ObjectScale enables you easily to manage globally distributed storage infrastructure under a single object store with anywhere access to content.

ObjectScale is built with specific design principles:

• Global namespace with eventual consistency
• Scale-out capabilities
• Secure multi-tenancy
• Superior performance for both small and large objects

The platform was built as a distributed system, following the microservices principle of cloud applications. ObjectScale has a layered architecture, in which each function in the system is built as an independent layer, making them horizontally scalable across all nodes and enabling high availability.

Deploy ObjectScale using the individual VMware software components:

• VMware Cloud Foundation with Tanzu
• NSX-T
• vSAN
• vSAN Data Persistence platform （optional）

VMware Cloud Foundation with Tanzu

VCF with Tanzu provides numerous options for running a Kubernetes environment. K8s allows organizations to develop and innovate applications quickly, but the storage infrastructure needs to be just as agile. When using modern stateful services, in conjunction with a shared nothing architecture that deploys built-in services such as replication and encryption, traditional storage consumption may need to change.

vSAN Data Persistence platform

vSAN DPp provides a framework for our service partners to integrate with the VMware Cloud Foundation infrastructure. vSAN DPp allows organizations to run stateful services that can quickly scale while simplifying operations and reducing costs. This allows you to deploy stateful services, in parallel with traditional applications, using the same infrastructure, with vCenter as a common management interface.

To learn more about running modern applications in your data center, here are some additional resources.

Resources:

• See the Solution Overview for details, including features and benefits
  • Obtain deployment guidance from the new VMware technical white paper: Dell ObjectScale on vSphere with Tanzu: Running modern applications in the traditional datacenter
• See Dell’s technical blog for insights on deploying the solution with Dell VxRail, hyperconverged infrastructure (HCI) built with and for VMware. Running Dell ObjectScale on VMware vSphere with Tanzu
• VMware: Modern Stateful Services
• ObjectScale Installation Guide for VMware vSphere
• ObjectScale Overview and Architecture
• ObjectScale Administrator Guide
• Visit the Dell ObjectScale website for additional ObjectScale resources.   

Author: Jarvis Zhu

Dell Technologies is excited to announce the availability of ECS 3.7, released on February 10, 2022. This release introduces the following new and incremental features to the ECS platform:

• S3 select 
• Deployable with SSD and HDD intermix nodes within a VDC
• Tokenization support for metadata search
• New hardware disks sizing options

These features continue to build upon the capabilities of the ECS platform and our broader object portfolio. Read on to learn more.

S3 select

One of the key features included with ECS 3.7 is S3 select. This feature enables applications to retrieve only a subset of data from an object by using simple SQL expressions. When you use S3 select to retrieve only the data needed by your application, you can achieve drastic performance increases and network bandwidth saving.

Let’s look at an example scenario with a 2 GB csv object. Without S3 select, your application would have to download the entire 2 GB object and then do the processing on that data. With s3 select, your application issues the SQL select command and gets only a small subset of that data.

You can use S3 select for objects in csv, json and parquet formats. It supports querying gzip/bzip2 compressed objects of the above 3 file types.

S3 select is commonly used by query engine, like presto. A connector like presto can determine if a particular query can be sent directly to the storage. For example: s3 select pushdown.

For AWS S3 compliant partial reads of an object that offloads, these reads query and sort to ECS rather than using compute resources. This feature may provide a performance benefit for use cases where network bandwidth or compute resources are a bottleneck.

S3 select is disabled by default on system with 64 GB memory and enabled by default on systems with 192 GB memory (EXF 900 only). 192 GB memory is a recommendation, however, to enable it on 64 GB memory, you must open a service request (SR) with Dell. We will notify you that memory usage will increase and may impact other operations, and then will enable the feature.

Deploy with SSD and HDD intermix nodes within a VDC

With ECS 3.7, you can create a VDC that consists of both EXF 900 nodes and any of the ECS HDD-based nodes (including EX300, EX500, EX3000 and Gen2 U-series). This ability allows for a single point of management for all node types. EXF 900 and HDD nodes must be in their own storage pools. You have the flexibility to add nodes of different media types to an existing VDC or new VDC per the following options:

• Fresh install with AFA/HDD intermix mode
• Extend node and add storage pool to support AFA/HDD intermix mode:
  • Extend HDD only VDC to coexisting VDC
  • Extend AFA only VDC to coexisting VDC

ECS 3.7 does not support replication between AFA and HDD nodes. However, you can use ECS Sync to move or copy data between the two storage pools. 

Note: ECS does not support a replication group (RG) containing both HDD and AFA storage pools.

Tokenization support for metadata search

Metadata search with tokenization enables you to search for objects that have a specific metadata value in an array of metadata values. You must choose this method when you create the bucket, and you can include the option when creating the bucket through the S3 create bucket API by including the header x-emc-metadata-search-tokens: true in the request.

For example, in the method without tokenization, with the command x-amzmeta-countries= [france,uk], the value france,uk is considered as a single value.

In the method with tokenization, with the command x-amzmeta-countries= [france,uk], the value france,uk is considered as two values separated with the delimiter.

New hardware disk-sizing options

Finally, ECS 3.7 introduces two new disk-sizing options, each with their own advantages:

• EX500 2 TB and 4 TB disk configurations give you options.
• EXF900 16 TB NVMe disk configuration provides more density, performance and agility.

In closing

The 3.7 release introduces several new and incremental features to the ECS platform, and reiterates the advantages of ECS and our broader object portfolio. 

For more information about the ECS, see the document ECS: Overview and Architecture and the ObjectScale and ECS Info Hub.

Author: Jarvis Zhu

Ransomware is defined as a form of malware that encrypts a victim’s files. The attacker will then demand a ransom from the victim and will only restore access after a payment has been made. These attackers are unscrupulous, always looking for opportunities to exploit weaknesses in potential victim’s defences. 

In the VMware 2020 Cybersecurity Outlook Report, defence evasion is a key tool for these attackers.  So having the right protection in place is paramount. Here are some highlights:

 A wiper attack involves wiping/overwriting/removing data from the victim. Unlike typical cyber-attacks that tend to be for monetary gain, wiper attacks are destructive in nature and often do not involve a ransom. Wiper malware can however be used to cover the tracks of a separate data theft.

Object storage can be regarded as a potential weak point in an organization’s armour. There are some key considerations about object storage that you need to be aware of when putting a security plan in place:

• Object storage platforms typically have no security monitoring tools to make you aware that your data is under threat. 
• Ransomware attackers usually target weak links within IT security and if they somehow obtain the secret key, they can gain access to petabytes of data with no security tools actively monitoring for these potential intrusions.
• Object is often used as a backup and that can make it a soft target because it’s not actively monitored.
• Also, Object can also be used for compliance data for legal hold, making it a target.
• With no notion of native, namespace level snapshots on object platforms recovery is made difficult.
• A few lines of python code are sufficient to attack object storage over S3.
• If your data is important you need to get monitoring in place before your data is attacked unknowingly. 

So how can I ensure my Object storage is safe and actively monitored? 

Protecting against any security threats including ransomware is a layered approach. Currently in Dell EMC Elastic Cloud Storage (ECS) you can use versioning to retain multiple copies of an object to protect against potential attacks.  For many years ECS has provided SEC17a-4(f) level compliance as a WORM-enabled capability when leveraging Amazon S3 API retention extensions.  This WORM capability has been expanded in ECS version 3.6.2 with the addition of S3 Object Lock.  Having these in place will give you a good foundation of protection for your object storage. 

Building on this to offer superior protection to our customers, we have partnered with Superna. With ECS and the Ransomware Defender from Superna, we can monitor user behaviour and detect potential threats to systems quickly. If potential threats do materialize, you can be alerted quickly to disable the user keys to mitigate the threat. Alternatively, you can configure Ransomware Defender to automatically lock the corresponding application user when it detects malicious activity. This can help expedite the recovery process by providing the user with a list of infected objects. The following figure shows a thorough workflow of how Superna can help secure your storage. 

For a demo of the functionality of this partnership, see Eyeglass Ransomware Defender for ECS Overview.

With this partnership we believe we can offer better protection for our customers and allow them to defend themselves against potential external security threats. 

Author: Finbarr O'Riordan  @finbarrorcork on Twitter

The data storage industry has three distinct pillars: block, file, and object. Traditionally, object storage has been leveraged for backup, long term archive, and data lake storage. It has been regarded as “cheap and deep”, but that perception is beginning to change as more and more application workloads suited to object storage are evolving. 

With the introduction of flash-based storage systems, object is now capable of handling high performance workloads at low latencies. Object use cases are expanding into other workloads such as Artificial Intelligence (AI), Deep Learning (DL), Machine Learning (ML), and DevOps. These analytical applications produce a wide range of data types and sizes. With object storage supporting unlimited fully customizable metadata, these features make it easier to locate data for use in AI and ML algorithms, offering enhanced insight for our customers.  

Exponential data growth in recent years presents economic challenges to companies. The scalable architecture of object storage means it’s well suited to handle data growth, allowing customers to add additional nodes when required. 

Elastic Cloud Storage (ECS), Dell EMC’s object storage system, is capable of scaling linearly based on capacity and performance. With flash-based storage, object is a viable alternative for addressing different business needs. Additionally, with the cost of flash storage falling and price points coming close to those of HDDs, the cost difference between the two technologies is narrowing. Customers can leverage higher performance and cost-effective object storage to gain the necessary insight from their data by using AI and ML to extract the intelligence they require to make strategic business decisions.

Over time, the S3 API has become the defacto standard for object storage. S3 allows for sharing of code, software, and tools -- which fits perfectly into the agile development world of DevOps. Having robust APIs is critical, because it allows customers to leverage automation and orchestration methods to respond quickly as required. Object storage APIs that are application and cloud centric allow you to deploy applications rapidly through code and processes. In certain industry verticals such as medical, financial, media, or automotive, flash object can be leveraged for research, financial modeling, streaming services, or CAD use cases. These are ideal for the type of applications that use massive data sets and high performing environments. 

In November 2020, we launched our flash object appliance: the Dell EMC EXF900. It is built with NVMe-based SSDs on Dell EMC PowerEdge servers, and leverages the NVMeOF (non-volatile memory express over fabrics) protocol for its backend network. This enables node-to-node communication and unlocks the true potential of the all-flash system’s throughput rate, especially in large scale deployments. 

With that, we are seeing incredible performance improvements in terms of bandwidth and transactions per second (TPS) when comparing the EXF900 to previous appliance releases. 

We believe the EXF900, with its enterprise performance levels, is perfectly suited to serve the changing workload needs of our customers. The modern data center is evolving and so are the workloads our customers are running on our systems. With its high performance and ability to automate easily, the EXF900 can deliver the scalability and agility they need. 

This significant combination of performance and security features ensures that ECS can support customers’ data-intensive workload requirements while protecting data from malicious activities.

Authors:

Finbarr O’Riordan  @finbarrorcork on Twitter

Christoffer Braendstrup

Dell EMC ECS 3.6.2, available for download since August 5, 2021, includes Object Lock support for our customers. This has been a popular ask and we are delighted to be able to deliver this to our Object Storage install base as it enables them to satisfy many use cases and help them in their daily roles. 

ECS allows you to store objects using a write-once-read-many (WORM) model through Object Lock. This feature prevents objects from being deleted or overwritten for a specified time or indefinitely. Also, Object Lock helps to meet WORM storage related regulatory requirements and adds a protection layer against object modifications and deletion.

ECS Object Lock allows you to manage object retention through retention periods and legal holds. With a retention period, you can specify a period during which an object remains locked. During the specified period, the object is WORM-protected, that is, the object cannot be overwritten or deleted. Legal hold provides the same protection as retention period but is independent from retention period, and does not have an expiration date. Legal hold is retained in objects until you explicitly remove it. Any user who has the appropriate Object Lock permissions can specify retention period and legal hold in objects. 

So, let’s look at a practical example for how we would use these. We may have a situation in a medical environment whereby patient files are not set up correctly for retention purposes, and we have a regulatory requirement to retain these files. To comply with government regulations, we can use the following command to put a legal hold on a bucket that contains the medical records.

S3curl.pl --id=ecsflex -- http://${s3ip} /my-bucket/obj?legalhold -X PUT –d “<LegalHold><Status>on/Status></LegalHold>”

After he places a legal hold on the necessary buckets, our trusty storage administrator should be prepared if an audit is held. 

Next let’s review how we use retention; Object lock has two retention modes:

1. Compliance -- This is regarded as the stricter of the two modes and is primarily targeted for regulatory compliance for certain customer use cases. Users cannot overwrite or delete an object version. Additionally, users can neither remove nor shorten an object retention. However, with s3: PutObjectRetention permission, you can increase an object’s retention period.   
2. Governance -- The governance protection mode is focused on protecting against potential security vulnerabilities such as rogue actors, accidental deletion, or comprised credentials. Ordinary users cannot overwrite or delete an object version, but users with the special privilege of s3:BypassGovernanceRetention can remove or shorten an object retention and delete locked objects. This is, in essence, a superuser privilege, so it is not granted lightly. Additionally, a user with the s3:PutObjectRetention permission can increase the object retention period. 

So, let’s look at a practical example for how we would use these modes. Let’s say from a governance perspective that we have an application owner who is working on an IT skunkworks type project that bore fruit, and they want to make sure that their work is protected and guards against any potential ransomware attack or through accidental deletion. To extend a retention time out to the year 2030 on an existing bucket, they can use this curl command.

S3curl.pl --id=ecsflex -- http://${s3ip} /my-bucket/obj?retention -X PUT –d “<Retention><Mode>GOVERNANCE</Mode><RetainUntilDate>2030-01-01T00:00:00.000Z</RetainUntilDate></Retention>”

This will ensure that the bucket is more secure and protects the user’s work from being overwritten.

ECS Object Lock fulfils some key requirements:

• Enables the management and enforcement of retention policies and legal holds for objects and buckets
• Supports a Governance and a Compliance version of enforcement
• Maintains data integrity and version consistency in multiple sites

We have delivered an API that enables customers to easily manage their Buckets and Objects while protecting themselves and complying to best practice standards. For more detail and other examples, please see our 3.6.2 Dell EMC ECS Data Access Guide.

Notes:

• The ECS Object Lock feature supports only the versioning enabled buckets. 
• There is no ECS user interface for Object Lock. It can be accessed through ECS Object Lock APIs. (In the 3.6.2 Dell EMC ECS Data Access Guide, for the Object Lock API examples, see the section “Object Lock API Examples”; for the list of supported S3 APIs, see the section “S3 API supported and unsupported features”.)
• The locked objects are protected from life cycle deletions.

Author: Finbarr O’Riordan  @finbarrorcork on Twitter

Dell EMC ECS supported WORM (write once, read many) based retention, starting with ECS 2.X. To provide more compatibility with more applications, ECS now supports the object lock feature (starting with ECS 3.6.2), which is compatible with the capabilities of Amazon S3 object lock.

Object lock is designed to meet compliance requirements such as SEC 17a4(f), FINRA Rule 4511(c), and CFTC Rule 17.

Object lock overview

Object lock prevents object version deletion during a user-defined retention period. Immutable S3 objects are protected using object- or bucket-level configuration of WORM and retention attributes. The retention policy is defined using the S3 API or bucket-level defaults. Objects are locked for the duration of the retention period, and legal hold scenarios are also supported.

There are two lock types for object lock:

• Retention period -- Specifies a fixed period of time during which an object version remains locked. During this period, your object version is WORM-protected and can't be overwritten or deleted.
• Legal hold -- Provides the same protection as a retention period, but it has no expiration date. Instead, a legal hold remains in place until you explicitly remove it. Legal holds are independent from retention periods.

There are two modes for the retention period:

• Governance mode -- users can't overwrite or delete an object version or alter its lock settings unless they have special permissions. With governance mode, you protect objects against being deleted by most users, but you can still grant some users permission to alter the retention settings or delete the object if necessary. You can also use governance mode to test retention-period settings before creating a compliance-mode retention period.
• Compliance mode -- a protected object version can't be overwritten or deleted by any user, including the root user in your account. When an object is locked in compliance mode, its retention mode can't be changed, and its retention period can't be shortened. Compliance mode helps ensure that an object version can't be overwritten or deleted for the duration of the retention period.

Object lock and lifecycle

Objects under lock are protected from lifecycle deletions.

Lifecycle logic is made difficult because of the variety of behavior of different locks. From a lifecycle point of view there are locks without a date, locks with date that can be extended, and locks with date that can be decreased.

• For compliance mode, the retain until date can't be decreased, but can be increased.
• For governance mode, the lock date can increase, decrease, or be removed.
• For legal hold, the lock is indefinite.

Some key points for the S3 object lock with ECS

• Object lock requires FS (File System) disabled on bucket in ECS version 3.6.2.
• Object lock requires ADO (Access During Outage) disabled on bucket in ECS version 3.6.2.
• Object lock is only supported by S3 API, not UI workflows in ECS version 3.6.2.
• Object lock only works with IAM, not legacy accounts.
• Object lock works only in versioned buckets.
• Enabling locking on the bucket automatically makes it versioned.
• Once bucket locking is enabled, it is not possible to disable object lock or suspend versioning for the bucket.
• A bucket has a default configuration including a retention mode (governance or compliance) and a retention period (which is days or years).
• Object locks apply to individual object versions only.
• Different versions of a single object can have different retention modes and periods.
• A lock prevents an object from being deleted or overwritten. Overwritten does not mean that new versions can't be created (new versions can be created with their own lock settings).
• An object can still be deleted. It creates a delete marker and the version still exists and is locked.
• Compliance mode is stricter, locks can't be removed, decreased, or downgraded to governance mode.
• Governance mode is less strict, it can be removed, bypassed, or elevated to compliance mode.
• An object can still be deleted, but the version still exists and is locked.
• Updating an object version's metadata, which occurs when you place or alter an object lock, doesn't overwrite the object version or reset its Last-Modified timestamp.
• Retention period can be placed on an object explicitly, or implicitly through a bucket default setting.
• Placing a default retention setting on a bucket doesn't place any retention settings on objects that already exist in the bucket.
• Changing a bucket's default retention period doesn't change the existing retention period for any objects in that bucket.
• Object lock and traditional bucket/object ECS retention can co-exist.

ECS object lock condition keys

Access control using IAM policies is an important part of the object lock functionality. The s3:BypassGovernanceRetention permission is important because it is required to delete a WORM-protected object in Governance mode.  IAM policy conditions have been defined below to allow you to limit what retention period and legal hold can be specified in objects.

ECS object lock API examples

This section lists s3curl examples of object Lock APIs. Put and Get object lock APIs can be used with and without the versionId parameter. If no versionId parameter is used, then the action applies to the latest version.

Conclusion

Dell EMC ECS object lock helps to protect object versions from accidental or malicious deletion, such as a ransomware attack. It does this by allowing object versions to enter a WORM state where access is restricted based on attributes set on the object version.