What's new in ECS 3.7
Tue, 21 Nov 2023 03:24:35 -0000
|Read Time: 0 minutes
Dell Technologies is excited to announce the availability of ECS 3.7, released on February 10, 2022. This release introduces the following new and incremental features to the ECS platform:
- S3 select
- Tokenization support for metadata search
- New hardware disks sizing options
These features continue to build upon the capabilities of the ECS platform and our broader object portfolio. Read on to learn more.
S3 select
One of the key features included with ECS 3.7 is S3 select. This feature enables applications to retrieve only a subset of data from an object by using simple SQL expressions. When you use S3 select to retrieve only the data needed by your application, you can achieve drastic performance increases and network bandwidth saving.
Let’s look at an example scenario with a 2 GB csv object. Without S3 select, your application would have to download the entire 2 GB object and then do the processing on that data. With s3 select, your application issues the SQL select command and gets only a small subset of that data.
You can use S3 select for objects in csv, json and parquet formats. It supports querying gzip/bzip2 compressed objects of the above 3 file types.
S3 select is commonly used by query engine, like presto. A connector like presto can determine if a particular query can be sent directly to the storage. For example: s3 select pushdown.
For AWS S3 compliant partial reads of an object that offloads, these reads query and sort to ECS rather than using compute resources. This feature may provide a performance benefit for use cases where network bandwidth or compute resources are a bottleneck.
S3 select is disabled by default on system with 64 GB memory and enabled by default on systems with 192 GB memory (EXF 900 only). 192 GB memory is a recommendation, however, to enable it on 64 GB memory, you must open a service request (SR) with Dell. We will notify you that memory usage will increase and may impact other operations, and then will enable the feature.
Tokenization support for metadata search
Metadata search with tokenization enables you to search for objects that have a specific metadata value in an array of metadata values. You must choose this method when you create the bucket, and you can include the option when creating the bucket through the S3 create bucket API by including the header x-emc-metadata-search-tokens: true in the request.
For example, in the method without tokenization, with the command x-amzmeta-countries= [france,uk], the value france,uk is considered as a single value.
In the method with tokenization, with the command x-amzmeta-countries= [france,uk], the value france,uk is considered as two values separated with the delimiter.
New hardware disk-sizing options
Finally, ECS 3.7 introduces two new disk-sizing options, each with their own advantages:
- EX500 2 TB and 4 TB disk configurations give you options.
- EXF900 16 TB NVMe disk configuration provides more density, performance and agility.
In closing
The 3.7 release introduces several new and incremental features to the ECS platform, and reiterates the advantages of ECS and our broader object portfolio.
For more information about the ECS, see the document ECS: Overview and Architecture and the ObjectScale and ECS Info Hub.
Author: Jarvis Zhu