Home > Storage > PowerScale (Isilon) > Product Documentation > Protocols > Dell EMC PowerScale OneFS S3 Overview > Objects
The AWS S3 data model is a flat structure—you create a bucket, and the bucket stores objects. There is no hierarchy of subbuckets or subfolders. However, AWS S3 provides a logical hierarchy using object key name prefixes and delimiters to support a concept of folders. For example, instead of naming an object sample.jpg in the bucket named examplebucket, you can name it photos/samples/sample.jpg. The photos/samples/ becomes the object key name prefix using the slash (/) as delimiter.
OneFS maps objects to files and creates directories as object key name prefixes implicitly. The OneFS file system requires the following rules for object naming:
Delimiter contains only one single slash(“/”) character by default.
Delimiter contains only one single slash(“/”) at the end, such as “abc/”.
Delimiter does not contain any slash(“/”), such as “abcd”. Therefore, “/abcd” and “ab/cd” are not valid delimiters.
The multipart upload allows users to upload new large files or make a copy of an existing file in parts for better uploading performance. After receiving the CreateMultipartUpload request from client, OneFS processes the multipart upload by writing each part to a temporary directory. It completes the multipart upload after all the parts are uploaded successfully.
Parts are uploaded to the temporary directory .isi_s3_parts_UploadId, and the temporary directory is created under the target directory. Upon receiving the complete multipart-upload request, OneFS concatenates the temporary files to the target file. You can access the file as object in your bucket. Starting from OneFS 9.3.0, MPU complete request is semi-instant if met following conditions:
Table 2 shows the supported S3 object operations in OneFS 9.0. See the document Dell EMC PowerScale: OneFS S3 API Guide on Dell.com/StorageResources for details about each supported API.
API name in AWS S3 API reference
API name in AWS S3 API reference | Note |
DeleteObject | Delete a single object from a bucket. Deleting multiple objects from a bucket using a single request is not supported. |
DeleteObjects | Starting from OneFS 9.3.0, delete multiple objects from a bucket using a single request is supported. |
GetObject | Retrieve an object content. |
GetObjectAcl | Get the ACL of an object. |
HeadObject | HEAD operation retrieves metadata from an object without returning the object itself. This operation is useful if you are only interested in an object's metadata. The operation returns a 200 OK if the object exists and if you have permission to access it. Otherwise, the operation might return responses such as 404 Not Found and 403 Forbidden. |
PutObject | Add an object to a bucket. |
CopyObject | Create a copy of an object that is already stored in OneFS. You can treat it as server-side-copy which reduces the network traffic between the clients and OneFS. |
PutObjectAcl | Set the ACL permissions for an object that exists in a bucket. |
CreateMultipartUpload | Initiate a multipart upload and return an upload ID. This upload ID is used to associate with all the parts in the specific multipart upload. You can specify this upload ID in each of your subsequent upload part requests. You also include this upload ID in the final request to either complete or cancel the multipart upload request. |
UploadPart | Upload a part in a multipart upload. Each part must be at least 5 MiB, except the last part. The maximum size of each part is 5 GiB. |
UploadPartCopy | Upload a part by copying data from an existing object in OneFS as the data source. Each part must be at least 5 MiB in size, except the last part. The maximum size of each part is 5 GiB. |
CompleteMultipartUpload | Complete a multipart upload by assembling previously uploaded parts. |
ListParts | List the parts that have been uploaded for a specific multipart upload. |
AbortMultipartUpload | Cancel a multipart upload. After a multipart upload is aborted, no additional parts can be uploaded using that upload ID. The storage consumed by any previously uploaded parts is freed. However, if any uploads of parts are in progress, those uploads might or might not succeed. As a result, it might be necessary to cancel a given multipart upload multiple times to free all storage consumed by all parts. |
AWS S3 may use an MD5 Checksum as an ETag value for objects. This value may be specified in the HTTP Header “Content-MD5”. In OneFS 9.0 and OneFS 9.1, OneFS uses the MD5 value from client as an ETag directly instead of calculating it by itself. If the MD5 is not specified in client request, OneFS generates a unique string for that file as an ETag in response. This behavior is different from AWS S3.
Most S3 applications do not send the MD5 value in their requests, thus, OneFS generates a unique string for that file as an ETag in response. This behavior causes many issues with applications that rely on the ETag value. Starting from OneFS 9.2, OneFS introduces two new options to allow administrators to specify if the MD5 should be calculated and verified.
These two options are under the S3 zone settings. You can configure them using CLI isi s3 settings zone modify --use-md5-for-etag=true/false --validate-content-md5=true/false or WebUI, shown in Figure 5.
# isi s3 settings zone view
Root Path: /ifs
Base Domain:
Object ACL Policy: replace
Bucket Directory Create Mode: 0777
Use Md5 For Etag: No
Validate Content Md5: No
| --use-md5-for-etag=false | --use-md5-for-etag=true |
--validate-content-md5=false | This is the default value. If “Content-MD5” exists in client request, OneFS uses it directly as the ETag without validation and checking the BASE64 encoding format. If “Content-MD5” does not exist in client request, OneFS generates a unique string for that file as the ETag. | If “Content-MD5” exists in client request and its value is properly encoded as BASE64 format, OneFS uses it as the ETag without validation. If “Content-MD5” does not exist in client request, OneFS calculates the MD5 value as the ETag. |
--validate-content-md5=true | If “Content-MD5” exists in client request and its value is properly encoded as BASE64 format, OneFS calculates the MD5 value and compare with the MD5 value from client request, if matched, uses it as the ETag. Otherwise, an error is returned to client. If “Content-MD5” does not exist in client request, OneFS generates a unique string for that file as the ETag. | If “Content-MD5” exists in client request, OneFS calculates the MD5 value and compare with the MD5 value from client request, if matched, uses it as the ETag. Otherwise, an error is returned to client. If “Content-MD5” does not exist in client request, OneFS calculates the MD5 value as the ETag. |
Note: Objects created with a multipart upload request do not use MD5 value as ETag.