What's new in ECS 3.7
Tue, 21 Nov 2023 03:24:35 -0000
|Read Time: 0 minutes
Dell Technologies is excited to announce the availability of ECS 3.7, released on February 10, 2022. This release introduces the following new and incremental features to the ECS platform:
- S3 select
- Tokenization support for metadata search
- New hardware disks sizing options
These features continue to build upon the capabilities of the ECS platform and our broader object portfolio. Read on to learn more.
S3 select
One of the key features included with ECS 3.7 is S3 select. This feature enables applications to retrieve only a subset of data from an object by using simple SQL expressions. When you use S3 select to retrieve only the data needed by your application, you can achieve drastic performance increases and network bandwidth saving.
Let’s look at an example scenario with a 2 GB csv object. Without S3 select, your application would have to download the entire 2 GB object and then do the processing on that data. With s3 select, your application issues the SQL select command and gets only a small subset of that data.
You can use S3 select for objects in csv, json and parquet formats. It supports querying gzip/bzip2 compressed objects of the above 3 file types.
S3 select is commonly used by query engine, like presto. A connector like presto can determine if a particular query can be sent directly to the storage. For example: s3 select pushdown.
For AWS S3 compliant partial reads of an object that offloads, these reads query and sort to ECS rather than using compute resources. This feature may provide a performance benefit for use cases where network bandwidth or compute resources are a bottleneck.
S3 select is disabled by default on system with 64 GB memory and enabled by default on systems with 192 GB memory (EXF 900 only). 192 GB memory is a recommendation, however, to enable it on 64 GB memory, you must open a service request (SR) with Dell. We will notify you that memory usage will increase and may impact other operations, and then will enable the feature.
Tokenization support for metadata search
Metadata search with tokenization enables you to search for objects that have a specific metadata value in an array of metadata values. You must choose this method when you create the bucket, and you can include the option when creating the bucket through the S3 create bucket API by including the header x-emc-metadata-search-tokens: true in the request.
For example, in the method without tokenization, with the command x-amzmeta-countries= [france,uk], the value france,uk is considered as a single value.
In the method with tokenization, with the command x-amzmeta-countries= [france,uk], the value france,uk is considered as two values separated with the delimiter.
New hardware disk-sizing options
Finally, ECS 3.7 introduces two new disk-sizing options, each with their own advantages:
- EX500 2 TB and 4 TB disk configurations give you options.
- EXF900 16 TB NVMe disk configuration provides more density, performance and agility.
In closing
The 3.7 release introduces several new and incremental features to the ECS platform, and reiterates the advantages of ECS and our broader object portfolio.
For more information about the ECS, see the document ECS: Overview and Architecture and the ObjectScale and ECS Info Hub.
Author: Jarvis Zhu
Related Blog Posts
What's new in ECS 3.8.0.1
Thu, 08 Dec 2022 15:40:00 -0000
|Read Time: 0 minutes
ECS 3.8.0.1 includes these new and updated features:
- Increased data mobility and flexibility with new data mobility feature
- Expanded external key management (KMIP) support
- Object Lock enhancements for ADO
- Security Token Service (STS) GetFederationToken support
- New hardware disk sizing option for EX500
- Memory upgrade expansion
Data mobility
Data mobility, also known as copy-to-cloud, is a new feature in version 3.8.0.1. With data mobility, a user can copy local object data to an external S3 target, such as a secondary ECS that is not federated or a public cloud target (currently AWS targets only).
Data mobility is configured as a bucket option in the UI, as shown in Figure 1. It can be monitored by an account admin or system admin within the UI. The admin can define policies about source and target buckets and criteria for objects. The admin can also monitor the logs for all copy operations at the object level, including the copy time, source object key, object size, target endpoint, duration, and result of the copy operation.
Figure 1 Data mobility configuration
We have also extended our ecosystem to support a multi-cloud experience for Snowflake, which runs on the AWS public cloud platform. Dell and Snowflake customers can use on-premises data stored on Dell ECS while keeping their data local or seamlessly copying it to public clouds and use Snowflake’s ecosystem of cloud-based data analysis services.
Expanded external key management support
The Key Management Interoperability Protocol (KMIP) is an extensible communication protocol that defines message formats for the manipulation of cryptographic keys on a key management server.
A new external key management cluster type to support Thales CipherTrust has been added since Gemalto SafeNet KeySecure will end of life on December 31, 2023. ECS customers who are using KeySecure can migrate to CipherTrust Manager.
Object Lock support with ADO enabled
ECS Object Lock enhancements for Access During Outage (ADO) have been added in ECS 3.8.0.1. Object Lock now supports ADO Read Only (RO) by default. For Read Write (RW) mode, ECS continues to deny setting Object Lock on ADO buckets by default. There are flags at the namespace and individual bucket level that require users to agree that they understand the risk of losing locked versions during TSO but would still like to enable this feature. Customers need to refer to the latest ECS Data Access Guide or ask the Dell support team to help enable Object Lock RW in ADO. Once flags are set on a bucket, they cannot be disabled.
The following table shows the Object Lock support matrix:
ECS version | Setting flags | Non-ADO | ADO RO | ADO RW |
3.6.2/3.7/partial upgrade
| Cannot set flags
| Yes | No | No |
Full upgrade to 3.8.0.1 | Set to not allowed (by default)
| Yes | Yes | No |
Set to allowed
| Yes | Yes | Yes |
STS GetFederationToken support
The GetFederationToken API is now part of Security Token Service (STS), along with AssumeRole and AssumeRoleWithSAML. It is called by the IAM user, and it returns a set of temporary security credentials (consisting of an access key ID, a secret access key, and a security token) for that user. This operation federates the user. A typical use is in a proxy application that gets temporary security credentials on behalf of distributed applications inside a corporate network. The ECS 3.8 Administration Guide on Dell Technologies Support provides more details.
New hardware disk sizing option
ECS version 3.8.0.1 extends disk sizing support. Customers can now select an ECS EX500 20 TB disk option.
Memory upgrade expansion
ECS version 3.8.0.1 supports memory upgrade expansion to 192 GB on EX300, EX500, EX3000, and Gen 2 platforms, with the support of Dell Professional Services. For more information, contact your ECS Customer Service representative.
Conclusion
ECS version 3.8.0.1 introduces new and updated platform features. It reiterates the core value proposition of ECS and our broader object portfolio, amplifying the benefits of the new capabilities in the 3.8.0.1 payload and related hardware updates.
For more information, see the ECS: Overview and Architecture White Paper on the ObjectScale and ECS Info Hub.
Protect Against Potential Ransomware Attacks on Object Storage
Wed, 06 Oct 2021 13:23:05 -0000
|Read Time: 0 minutes
Ransomware is defined as a form of malware that encrypts a victim’s files. The attacker will then demand a ransom from the victim and will only restore access after a payment has been made. These attackers are unscrupulous, always looking for opportunities to exploit weaknesses in potential victim’s defences.
In the VMware 2020 Cybersecurity Outlook Report, defence evasion is a key tool for these attackers. So having the right protection in place is paramount. Here are some highlights:
A wiper attack involves wiping/overwriting/removing data from the victim. Unlike typical cyber-attacks that tend to be for monetary gain, wiper attacks are destructive in nature and often do not involve a ransom. Wiper malware can however be used to cover the tracks of a separate data theft.
Object storage can be regarded as a potential weak point in an organization’s armour. There are some key considerations about object storage that you need to be aware of when putting a security plan in place:
- Object storage platforms typically have no security monitoring tools to make you aware that your data is under threat.
- Ransomware attackers usually target weak links within IT security and if they somehow obtain the secret key, they can gain access to petabytes of data with no security tools actively monitoring for these potential intrusions.
- Object is often used as a backup and that can make it a soft target because it’s not actively monitored.
- Also, Object can also be used for compliance data for legal hold, making it a target.
- With no notion of native, namespace level snapshots on object platforms recovery is made difficult.
- A few lines of python code are sufficient to attack object storage over S3.
- If your data is important you need to get monitoring in place before your data is attacked unknowingly.
So how can I ensure my Object storage is safe and actively monitored?
Protecting against any security threats including ransomware is a layered approach. Currently in Dell EMC Elastic Cloud Storage (ECS) you can use versioning to retain multiple copies of an object to protect against potential attacks. For many years ECS has provided SEC17a-4(f) level compliance as a WORM-enabled capability when leveraging Amazon S3 API retention extensions. This WORM capability has been expanded in ECS version 3.6.2 with the addition of S3 Object Lock. Having these in place will give you a good foundation of protection for your object storage.
Building on this to offer superior protection to our customers, we have partnered with Superna. With ECS and the Ransomware Defender from Superna, we can monitor user behaviour and detect potential threats to systems quickly. If potential threats do materialize, you can be alerted quickly to disable the user keys to mitigate the threat. Alternatively, you can configure Ransomware Defender to automatically lock the corresponding application user when it detects malicious activity. This can help expedite the recovery process by providing the user with a list of infected objects. The following figure shows a thorough workflow of how Superna can help secure your storage.
For a demo of the functionality of this partnership, see Eyeglass Ransomware Defender for ECS Overview.
With this partnership we believe we can offer better protection for our customers and allow them to defend themselves against potential external security threats.
Author: Finbarr O'Riordan @finbarrorcork on Twitter