Home > Storage > PowerScale (Isilon) > Product Documentation > Protocols > Dell PowerScale: OneFS S3 API Guide > Additional features
AWS S3 may use an MD5 Checksum as an ETag value for objects. This value may be specified in the HTTP Header “Content-MD5”. In OneFS 9.0 and OneFS 9.1, OneFS uses the MD5 value from client as an ETag directly instead of calculating it by itself. If the MD5 is not specified in client request, OneFS generates a unique string for that file as an ETag in response. This behavior is different from AWS S3.
Most S3 applications do not send the MD5 value in their requests, thus, OneFS generates a unique string for that file as an ETag in response. This behavior causes many issues with applications that rely on the ETag value. Therefore, starting from OneFS 9.2, OneFS introduces two new options to allow administrators to specify if the MD5 should be calculated and verified.
These two options are under the S3 zone settings. You can configure them using CLI isi s3 settings zone modify --use-md5-for-etag=true/false --validate-content-md5=true/false or WebUI, shown as Figure 1.
# isi s3 settings zone view
Root Path: /ifs
Base Domain:
Object ACL Policy: replace
Bucket Directory Create Mode: 0777
Use Md5 For Etag: No
Validate Content Md5: No
Table 25 shows the different behavior by using the two new options.
| --use-md5-for-etag=false | --use-md5-for-etag=true |
--validate-content-md5=false | This is the default value. If “Content-MD5” exists in client request, OneFS uses it directly as the ETag without validation and checking the BASE64 encoding format. If “Content-MD5” does not exist in client request, OneFS generates a unique string for that file as the ETag. | If “Content-MD5” exists in client request and its value is properly encoded as BASE64 format, OneFS uses it as the ETag without validation. If “Content-MD5” does not exist in client request, OneFS calculates the MD5 value as the ETag. |
--validate-content-md5=true | If “Content-MD5” exists in client request and its value is properly encoded as BASE64 format, OneFS calculates the MD5 value and compare with the MD5 value from client request, if matched, uses it as the ETag. Otherwise, an error is returned to client. If “Content-MD5” does not exist in client request, OneFS generates a unique string for that file as the ETag. | If “Content-MD5” exists in client request, OneFS calculates the MD5 value and compare with the MD5 value from client request, if matched, uses it as the ETag. Otherwise, an error is returned to client. If “Content-MD5” does not exist in client request, OneFS calculates the MD5 value as the ETag. |
Note: Objects created with a multipart upload request do not use MD5 value as ETag.
A presigned URL gives you access to the object identified in the URL, OneFS supports presigned URLs to allow users to access objects without needing credentials. Please refer to AWS S3 Presigned URLs for more details.
There are two types of uploading options when authenticating requests using the Authorization header of AWS Signature Version 4:
Starting with OneFS 9.3.0, the chunked upload is introduced. With chunked upload, you can break up your payload into chunks. These can be fixed or variable-size chunks. By uploading data in chunks, you avoid reading the entire payload to calculate the signature. Instead, for the first chunk, you calculate a seed signature that uses only the request headers. The second chunk contains the signature for the first chunk, and each subsequent chunk contains the signature for the chunk that precedes it. At the end of the upload, you send a final chunk with 0 bytes of data that contains the signature of the last chunk of the payload. You can refer to AWS S3 Chunked Upload for more details.
Starting from OneFS 9.3.0, the concept of "inter-level directory" is introduced. An inter-level directory in OneFS refers to directories within an S3 bucket that contain the .isi_s3_dir file.
When uploading an object that requires a parent directory, such as uploading an object with the key "a/b/c" and directories "a/" and "b/" do not exist in OneFS, OneFS will automatically create these directories to store the file "c". Additionally, OneFS will create a file named .isi_s3_dir in each newly created directory.
The introduction of inter-level directories changes the OneFS behavior of the delete bucket and list objects S3 API.
| Before OneFS 9.3.0 | OneFS 9.3.0 and above |
Delete bucket | Bucket can be deleted only if the bucket path does not contain any directories and files. | Bucket can be deleted if the bucket path is empty or only contains the inter-level directories. The empty inter-level directories are deleted as well. |
List objects | List objects API lists empty directories as object. For example, “a/b/c/” is returned as object where “c/” is a OneFS directory. | List objects ignores empty inter-level directories without specify delimiter "/" . |
# tree -a /ifs/data/s3-bkt/
/ifs/data/s3-bkt/
0 directory, 0 files
# tree -a /ifs/data/s3-bkt/
/ifs/data/s3-bkt/
└── dir01
1 directory, 0 files
# tree -a /ifs/data/s3-bkt/
/ifs/data/s3-bkt/
└── dir01
└── .isi_s3_dir
1 directory, 1 file
In AWS S3, the object keys "a/" and "a/b" represent distinct data entities. AWS S3 permits users to delete the object "a/" while retaining the object "a/b".
However, within OneFS, if you explicitly upload two objects "a/" and "a/b", the object "a/" functions as a directory in OneFS, while "a/b" is recognized as a file nested under the directory "a/". Attempting to delete the object "a/" in OneFS fails with HTTP error 200 because the directory "a/" is not empty.
To align OneFS S3 behavior more closely with AWS S3, starting from OneFS 9.7.0, when deleting a directory object, the directory transitions into an inter-level directory with the creation of a .isi_s3_dir file. This transition occurs seamlessly without triggering any error prompts for clients. Below is an example: