Short articles related to Dell EMC ECS.
Dell ObjectScale on VMware Cloud Foundation with Tanzu
Mon, 27 Jun 2022 21:50:51 -0000
|Read Time: 0 minutes
Dell Technologies has engineered software-defined, enterprise-class, high performance, containerized architecture object storage in the VMware Tanzu platform called ObjectScale.
VMware and Dell Technologies have worked together to provide a framework for cloud-native object storage in vSphere, using the vSAN Data Persistence platform: Dell ObjectScale on VMware Cloud Foundation with Tanzu. Dell ObjectScale provides native Kubernetes, S3-compatible storage utilizing vSAN. ObjectScale integrates with vSphere, simplifying management for administrators. With rich S3 compatibility and self-service APIs, you can quickly spin up object storage containers to fuel applications, ranging from big data and analytics to ephemeral dev or test sandboxes.
Dell ObjectScale allows organizations to deliver scalable cloud services with the reliability and control of private cloud infrastructure like VMware. ObjectScale enables you easily to manage globally distributed storage infrastructure under a single object store with anywhere access to content.
ObjectScale is built with specific design principles:
The platform was built as a distributed system, following the microservices principle of cloud applications. ObjectScale has a layered architecture, in which each function in the system is built as an independent layer, making them horizontally scalable across all nodes and enabling high availability.
Deploy ObjectScale using the individual VMware software components:
VCF with Tanzu provides numerous options for running a Kubernetes environment. K8s allows organizations to develop and innovate applications quickly, but the storage infrastructure needs to be just as agile. When using modern stateful services, in conjunction with a shared nothing architecture that deploys built-in services such as replication and encryption, traditional storage consumption may need to change.
vSAN DPp provides a framework for our service partners to integrate with the VMware Cloud Foundation infrastructure. vSAN DPp allows organizations to run stateful services that can quickly scale while simplifying operations and reducing costs. This allows you to deploy stateful services, in parallel with traditional applications, using the same infrastructure, with vCenter as a common management interface.
To learn more about running modern applications in your data center, here are some additional resources.
Author: Jarvis Zhu
What's new in ECS 3.7
Wed, 09 Mar 2022 14:43:12 -0000
|Read Time: 0 minutes
Dell Technologies is excited to announce the availability of ECS 3.7, released on February 10, 2022. This release introduces the following new and incremental features to the ECS platform:
These features continue to build upon the capabilities of the ECS platform and our broader object portfolio. Read on to learn more.
One of the key features included with ECS 3.7 is S3 select. This feature enables applications to retrieve only a subset of data from an object by using simple SQL expressions. When you use S3 select to retrieve only the data needed by your application, you can achieve drastic performance increases and network bandwidth saving.
Let’s look at an example scenario with a 2 GB csv object. Without S3 select, your application would have to download the entire 2 GB object and then do the processing on that data. With s3 select, your application issues the SQL select command and gets only a small subset of that data.
You can use S3 select for objects in csv, json and parquet formats. It supports querying gzip/bzip2 compressed objects of the above 3 file types.
S3 select is commonly used by query engine, like presto. A connector like presto can determine if a particular query can be sent directly to the storage. For example: s3 select pushdown.
For AWS S3 compliant partial reads of an object that offloads, these reads query and sort to ECS rather than using compute resources. This feature may provide a performance benefit for use cases where network bandwidth or compute resources are a bottleneck.
S3 select is disabled by default on system with 64 GB memory and enabled by default on systems with 192 GB memory (EXF 900 only). 192 GB memory is a recommendation, however, to enable it on 64 GB memory, you must open a service request (SR) with Dell. We will notify you that memory usage will increase and may impact other operations, and then will enable the feature.
With ECS 3.7, you can create a VDC that consists of both EXF 900 nodes and any of the ECS HDD-based nodes (including EX300, EX500, EX3000 and Gen2 U-series). This ability allows for a single point of management for all node types. EXF 900 and HDD nodes must be in their own storage pools. You have the flexibility to add nodes of different media types to an existing VDC or new VDC per the following options:
ECS 3.7 does not support replication between AFA and HDD nodes. However, you can use ECS Sync to move or copy data between the two storage pools.
Note: ECS does not support a replication group (RG) containing both HDD and AFA storage pools.
Metadata search with tokenization enables you to search for objects that have a specific metadata value in an array of metadata values. You must choose this method when you create the bucket, and you can include the option when creating the bucket through the S3 create bucket API by including the header x-emc-metadata-search-tokens: true in the request.
For example, in the method without tokenization, with the command x-amzmeta-countries= [france,uk], the value france,uk is considered as a single value.
In the method with tokenization, with the command x-amzmeta-countries= [france,uk], the value france,uk is considered as two values separated with the delimiter.
Finally, ECS 3.7 introduces two new disk-sizing options, each with their own advantages:
The 3.7 release introduces several new and incremental features to the ECS platform, and reiterates the advantages of ECS and our broader object portfolio.
For more information about the ECS, see the document ECS: Overview and Architecture and the ObjectScale and ECS Info Hub.
Author: Jarvis Zhu
Protect Against Potential Ransomware Attacks on Object Storage
Wed, 06 Oct 2021 13:23:05 -0000
|Read Time: 0 minutes
Ransomware is defined as a form of malware that encrypts a victim’s files. The attacker will then demand a ransom from the victim and will only restore access after a payment has been made. These attackers are unscrupulous, always looking for opportunities to exploit weaknesses in potential victim’s defences.
In the VMware 2020 Cybersecurity Outlook Report, defence evasion is a key tool for these attackers. So having the right protection in place is paramount. Here are some highlights:
A wiper attack involves wiping/overwriting/removing data from the victim. Unlike typical cyber-attacks that tend to be for monetary gain, wiper attacks are destructive in nature and often do not involve a ransom. Wiper malware can however be used to cover the tracks of a separate data theft.
Object storage can be regarded as a potential weak point in an organization’s armour. There are some key considerations about object storage that you need to be aware of when putting a security plan in place:
So how can I ensure my Object storage is safe and actively monitored?
Protecting against any security threats including ransomware is a layered approach. Currently in Dell EMC Elastic Cloud Storage (ECS) you can use versioning to retain multiple copies of an object to protect against potential attacks. For many years ECS has provided SEC17a-4(f) level compliance as a WORM-enabled capability when leveraging Amazon S3 API retention extensions. This WORM capability has been expanded in ECS version 3.6.2 with the addition of S3 Object Lock. Having these in place will give you a good foundation of protection for your object storage.
Building on this to offer superior protection to our customers, we have partnered with Superna. With ECS and the Ransomware Defender from Superna, we can monitor user behaviour and detect potential threats to systems quickly. If potential threats do materialize, you can be alerted quickly to disable the user keys to mitigate the threat. Alternatively, you can configure Ransomware Defender to automatically lock the corresponding application user when it detects malicious activity. This can help expedite the recovery process by providing the user with a list of infected objects. The following figure shows a thorough workflow of how Superna can help secure your storage.
For a demo of the functionality of this partnership, see Eyeglass Ransomware Defender for ECS Overview.
With this partnership we believe we can offer better protection for our customers and allow them to defend themselves against potential external security threats.
Author: Finbarr O'Riordan @finbarrorcork on Twitter
All Flash Object Storage: Dell EMC ECS EXF900 Appliances Support New Workloads with Outstanding Performance
Fri, 24 Sep 2021 13:34:58 -0000
|Read Time: 0 minutes
The data storage industry has three distinct pillars: block, file, and object. Traditionally, object storage has been leveraged for backup, long term archive, and data lake storage. It has been regarded as “cheap and deep”, but that perception is beginning to change as more and more application workloads suited to object storage are evolving.
With the introduction of flash-based storage systems, object is now capable of handling high performance workloads at low latencies. Object use cases are expanding into other workloads such as Artificial Intelligence (AI), Deep Learning (DL), Machine Learning (ML), and DevOps. These analytical applications produce a wide range of data types and sizes. With object storage supporting unlimited fully customizable metadata, these features make it easier to locate data for use in AI and ML algorithms, offering enhanced insight for our customers.
Exponential data growth in recent years presents economic challenges to companies. The scalable architecture of object storage means it’s well suited to handle data growth, allowing customers to add additional nodes when required.
Elastic Cloud Storage (ECS), Dell EMC’s object storage system, is capable of scaling linearly based on capacity and performance. With flash-based storage, object is a viable alternative for addressing different business needs. Additionally, with the cost of flash storage falling and price points coming close to those of HDDs, the cost difference between the two technologies is narrowing. Customers can leverage higher performance and cost-effective object storage to gain the necessary insight from their data by using AI and ML to extract the intelligence they require to make strategic business decisions.
Over time, the S3 API has become the defacto standard for object storage. S3 allows for sharing of code, software, and tools -- which fits perfectly into the agile development world of DevOps. Having robust APIs is critical, because it allows customers to leverage automation and orchestration methods to respond quickly as required. Object storage APIs that are application and cloud centric allow you to deploy applications rapidly through code and processes. In certain industry verticals such as medical, financial, media, or automotive, flash object can be leveraged for research, financial modeling, streaming services, or CAD use cases. These are ideal for the type of applications that use massive data sets and high performing environments.
In November 2020, we launched our flash object appliance: the Dell EMC EXF900. It is built with NVMe-based SSDs on Dell EMC PowerEdge servers, and leverages the NVMeOF (non-volatile memory express over fabrics) protocol for its backend network. This enables node-to-node communication and unlocks the true potential of the all-flash system’s throughput rate, especially in large scale deployments.
With that, we are seeing incredible performance improvements in terms of bandwidth and transactions per second (TPS) when comparing the EXF900 to previous appliance releases.
We believe the EXF900, with its enterprise performance levels, is perfectly suited to serve the changing workload needs of our customers. The modern data center is evolving and so are the workloads our customers are running on our systems. With its high performance and ability to automate easily, the EXF900 can deliver the scalability and agility they need.
This significant combination of performance and security features ensures that ECS can support customers’ data-intensive workload requirements while protecting data from malicious activities.
Authors:
Finbarr O’Riordan @finbarrorcork on Twitter
Christoffer Braendstrup
Manage Object Retention with ECS Object Lock
Wed, 08 Sep 2021 20:24:39 -0000
|Read Time: 0 minutes
Dell EMC ECS 3.6.2, available for download since August 5, 2021, includes Object Lock support for our customers. This has been a popular ask and we are delighted to be able to deliver this to our Object Storage install base as it enables them to satisfy many use cases and help them in their daily roles.
ECS allows you to store objects using a write-once-read-many (WORM) model through Object Lock. This feature prevents objects from being deleted or overwritten for a specified time or indefinitely. Also, Object Lock helps to meet WORM storage related regulatory requirements and adds a protection layer against object modifications and deletion.
ECS Object Lock allows you to manage object retention through retention periods and legal holds. With a retention period, you can specify a period during which an object remains locked. During the specified period, the object is WORM-protected, that is, the object cannot be overwritten or deleted. Legal hold provides the same protection as retention period but is independent from retention period, and does not have an expiration date. Legal hold is retained in objects until you explicitly remove it. Any user who has the appropriate Object Lock permissions can specify retention period and legal hold in objects.
So, let’s look at a practical example for how we would use these. We may have a situation in a medical environment whereby patient files are not set up correctly for retention purposes, and we have a regulatory requirement to retain these files. To comply with government regulations, we can use the following command to put a legal hold on a bucket that contains the medical records.
S3curl.pl --id=ecsflex -- http://${s3ip} /my-bucket/obj?legalhold -X PUT –d “<LegalHold><Status>on/Status></LegalHold>”
After he places a legal hold on the necessary buckets, our trusty storage administrator should be prepared if an audit is held.
Next let’s review how we use retention; Object lock has two retention modes:
So, let’s look at a practical example for how we would use these modes. Let’s say from a governance perspective that we have an application owner who is working on an IT skunkworks type project that bore fruit, and they want to make sure that their work is protected and guards against any potential ransomware attack or through accidental deletion. To extend a retention time out to the year 2030 on an existing bucket, they can use this curl command.
S3curl.pl --id=ecsflex -- http://${s3ip} /my-bucket/obj?retention -X PUT –d “<Retention><Mode>GOVERNANCE</Mode><RetainUntilDate>2030-01-01T00:00:00.000Z</RetainUntilDate></Retention>”
This will ensure that the bucket is more secure and protects the user’s work from being overwritten.
ECS Object Lock fulfils some key requirements:
We have delivered an API that enables customers to easily manage their Buckets and Objects while protecting themselves and complying to best practice standards. For more detail and other examples, please see our 3.6.2 Dell EMC ECS Data Access Guide.
Notes:
Author: Finbarr O’Riordan @finbarrorcork on Twitter
Better Protection with Dell EMC ECS Object Lock
Tue, 31 Aug 2021 20:46:06 -0000
|Read Time: 0 minutes
Dell EMC ECS supported WORM (write once, read many) based retention, starting with ECS 2.X. To provide more compatibility with more applications, ECS now supports the object lock feature (starting with ECS 3.6.2), which is compatible with the capabilities of Amazon S3 object lock.
Object lock is designed to meet compliance requirements such as SEC 17a4(f), FINRA Rule 4511(c), and CFTC Rule 17.
Object lock prevents object version deletion during a user-defined retention period. Immutable S3 objects are protected using object- or bucket-level configuration of WORM and retention attributes. The retention policy is defined using the S3 API or bucket-level defaults. Objects are locked for the duration of the retention period, and legal hold scenarios are also supported.
There are two lock types for object lock:
There are two modes for the retention period:
Objects under lock are protected from lifecycle deletions.
Lifecycle logic is made difficult because of the variety of behavior of different locks. From a lifecycle point of view there are locks without a date, locks with date that can be extended, and locks with date that can be decreased.
Access control using IAM policies is an important part of the object lock functionality. The s3:BypassGovernanceRetention permission is important because it is required to delete a WORM-protected object in Governance mode. IAM policy conditions have been defined below to allow you to limit what retention period and legal hold can be specified in objects.
Condition Key | Description |
s3:object-lock-legal-hold | Enables enforcement of the specified object legal hold status |
s3:object-lock-mode | Enables enforcement of the specified object retention mode |
s3:object-lock-retain-until-date | Enables enforcement of a specific retain-until-date |
s3:object-lock-remaining-retention-days | Enables enforcement of an object relative to the remaining retention days |
This section lists s3curl examples of object Lock APIs. Put and Get object lock APIs can be used with and without the versionId parameter. If no versionId parameter is used, then the action applies to the latest version.
Operation | API request examples |
Create lock-enabled bucket | s3curl.pl --id=ecsflex --createBucket -- http://${s3ip}/mybucket -H "x-amz-bucket-object-lock-enabled: true" |
Enable object lock on existing bucket | s3curl.pl --id=ecsflex -- http://${s3ip}/my-bucket?enable-objectlock -X PUT |
Get bucket default lock configuration | s3curl.pl --id=ecsflex -- http://${s3ip}/my-bucket?object-lock |
Put bucket default lock configuration | s3curl.pl --id=ecsflex -- http://${s3ip}/my-bucket?object-lock -X PUT \ -d "<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ ObjectLockEnabled> <Rule><DefaultRetention><Mode>GOVERNANCE</Mode><Days>1</Days></ DefaultRetention></Rule></ObjectLockConfiguration>" |
Get legal hold | s3curl.pl --id=ecsflex -- http://${s3ip}/my-bucket/obj?legal-hold |
Put legal hold on create | s3curl.pl --id=ecsflex --put=/root/100b.file -- http://${s3ip}/ my-bucket/obj -H "x-amz-object-lock-legal-hold: ON" |
Put legal hold on existing object | s3curl.pl --id=ecsflex -- http://${s3ip}/my-bucket/obj?legalhold -X PUT -d "<LegalHold><Status>OFF</Status></LegalHold>" |
Get retention | s3curl.pl --id=ecsflex -- http://${s3ip}/my-bucket/obj?retention |
Put retention on create | s3curl.pl --id=ecsflex --put=/root/100b.file -- http://${s3ip}/ my-bucket/obj -H "x-amz-object-lock-mode: GOVERNANCE" -H "x-amz-object-lock-retain-until-date: 2030-01-01T00:00:00.000Z" |
Put retention on existing object | s3curl.pl --id=ecsflex -- http://${s3ip}/my-bucket/obj? retention -X PUT -d "<Retention><Mode>GOVERNANCE</ Mode><RetainUntilDate>2030-01-01T00:00:00.000Z</ RetainUntilDate></Retention>" |
Put retention on existing object (with bypass) | s3curl.pl --id=ecsflex -- http://${s3ip}/my-bucket/obj? retention -X PUT -d "<Retention><Mode>GOVERNANCE</ Mode><RetainUntilDate>2030-01-01T00:00:00.000Z</ RetainUntilDate></Retention>" -H "x-amz-bypass-governance-retention: true" |
Dell EMC ECS object lock helps to protect object versions from accidental or malicious deletion, such as a ransomware attack. It does this by allowing object versions to enter a WORM state where access is restricted based on attributes set on the object version.