Dell PowerScale and Marvel Partner to Create Optimal Media Workflows
Tue, 01 Aug 2023 17:03:47 -0000
|Read Time: 0 minutes
Now in its 9th generation, Emmy-award-winning Dell PowerScale storage has been field proven in media workflows for over two decades and is the world’s most flexible1, efficient2, and secure3 scale-out NAS solution.
Our partnership with Marvel Studios is a wonderful example of the innovations we collaborate on with leading media and entertainment companies around the world—with PowerScale as the preeminent storage solution that enables data-driven workflows to accelerate content-creation pipelines.
Hear about Marvel Studios’ implementation of PowerScale directly in this educational video series from The Advanced Imaging Society:
The PowerScale OneFS advantage
The underlying OneFS file system leverages the foundations of clustered high-performance computing to solve the challenges of data protection at scale and client accessibility in a massively parallelized way. In practice, a single namespace that easily scales out with nodes to increase performance and capacity is a fundamentally game-changing architecture.
Media workflows require increased levels of access for the applications and users to provide for workflow collaboration in balance with security that doesn’t impede performance. Further, performance and access can’t be impeded even during hardware failures such as drive rebuilds or system upgrades to ensure that production work can continue uninterrupted while maintenance is being performed in the background.
Maximizing uptime correlates with fundamental business needs including meeting project timelines and budgets while ensuring that personnel have access to the content at the required performance levels, even during a background maintenance activity.
As a sufficiently advanced enterprise-class solution, PowerScale incorporates these capabilities to eliminate complexity and provide for increased uptime through its self-healing and self-managing functionality. (For more information, see the PowerScale OneFS Technical Overview.) This takes many of the traditional storage management burdens off the administrator’s plate, lowering the overhead and time needed to maintain storage, which is often increasing in size and scale.
While the benefits of collaboration over Ethernet-based storage are inherent in PowerScale, the user experience is also paramount in correlation with the performance of the network and underlying storage system. Operations such as playout and scrubbing need to perform reliably with no frame drops while providing response times that look and feel equivalent to working from the local workstation.
As I’ve participated in the development of media storage solutions from SCSI, Fibre Channel, iSCSI, SAN, and Ethernet, I’ve been able to test and compare solutions over the years and have closely watched the evolution and trends of these protocols in relation to their ability to support media workflows.
In 2007, I demonstrated real-time color grading with uncompressed content over 1 Gb Ethernet, the first of its kind. At the time, using Ethernet-based storage for color grading was largely unheard of, with few applications supporting it. That was more of an exercise to showcase the art of the possible in comparison with Fibre Channel-based solutions. The wider adoption of Ethernet for this particular use case was not yet of high interest because Ethernet speeds still needed to evolve. However, 1 Gb Ethernet was very appropriate for compressed media workflows and rendering, which were well aligned with the high-performance, scale-out design of PowerScale.
As 10 Gb Ethernet speeds became prevalent, there was a significant uptick in the adoption of Ethernet-based storage compared to Fibre Channel-based solutions for media use cases. I also started to see more datasets being moved over Ethernet rather than by sneaker net, physically delivering drives and tapes between locations. This led to cost and time savings for project timelines and budgets, among other benefits.
Fast-forward to 2014, when, with the OneFS 7.1.1 version supporting SMB multi-channel, we were able to use two 10 Gb Ethernet connections to support a stream of full resolution uncompressed 4K, whereas a single 10 Gb connection was only capable of supporting 2K full resolution streams. This began an adoption trend of Ethernet solutions for 4K full-resolution workflows.
In 2017, with the release of the F800 All-Flash PowerScale and OneFS 8.1, 40 Gb Ethernet speeds were supported. The floodgates were unlocked for media workflows. Multiple full-resolution 2K and 4K workflows could run on a single shared OneFS namespace with uncompromised performance. Workload consolidation could be performed and started to eliminate the need for multiple discreet storage solutions that were each supporting different parts of the pipeline, bringing all those together under a single unified OneFS namespace to streamline environments.
Complete pipeline transformations were taking place and began to replace iSCSI and Fibre Channel-based solutions at an accelerated pace, as those solutions were siloed within workgroups and inflexible with the emerging needs of collaboration. When the PowerScale F900 NVMe solution supporting 100 Gb Ethernet came out in 2021, the technology was set to change the industry yet again.
With the increasing prevalence of 100 Gb Ethernet over these past few years, performance parity with Fibre Channel-based solutions to support full-resolution 4K, 8K, and all related media workflows in between is no longer in question. Native Ethernet-based solutions are preferred for many reasons—including cloud capability, scale, cost, and supportability—to facilitate unstructured media datasets, leveraging the abundance of network engineering talent in comparison to Fibre Channel-trained engineers.
With reliability, performance, and shared access for collaboration delivering uncompromised benefits, we now look to several PowerScale storage capabilities that enable rich media ecosystems to be further streamlined and flourish.
There are four additional key areas of focus and their underlying feature sets that are increasingly important to today’s media ecosystems. They encompass:
- Security, physical and logical
- API orchestration
- Data movement
- Quality of service
Security
- In relation to PowerScale being the world’s most secure scale-out NAS solution and in alignment with the Trusted Partner Network (TPN), the OneFS operating system meets or exceeds the Motion Picture Association (MPA) Content Security Best Practices (CSBP) in all relevant areas of the Content Security Model (CSM).4
- Further, I’m seeing greater adoption of self-encrypting drives (SEDs), which provide encryption at the physical layer. Security auditing and multi-factor authentication are among the features being employed to protect the logical layer. Specifically, auditing has been available and used for many years now to provide a range of benefits beyond security.
- The real-time audit logs can be parsed to provide performance introspection, analysis, and user-trend insights in addition to identifying abnormal data-access patterns. The logs can also be used to correlate with levels of access to specific projects and files, correlating back to business-level insights and reports as well.
- I’m also keen on mentioning the OneFS embedded firewall, which provides connection-level protection at the storage network layer. Firewalls are typically employed in front of the storage or further upstream on the network, so having an additional firewall within the storage that protects the network ports on the storage itself is a powerful layer of security.
For more information about OneFS security, see Dell PowerScale OneFS: Security Considerations.
Data orchestration
- Data orchestration is paramount to workflow automation. If aspects of workflows can be automated and don’t require an operator to make a decision, they should be automated to remove the possibility for operator error and streamline the environment to accelerate workflows where possible.
- Orchestration is enabled through API calls to integrate PowerScale with the environment’s application layers, which can take mundane repeatable tasks off the operator’s plate and increase workflow efficiency.
For more information about PowerScale data orchestration, see the OneFS documentation on Dell Support.
Data movement
- Data movement is integral to media workflows today, and for that we look to the high-performance, highly reliable SyncIQ protocol embedded in PowerScale OneFS. SyncIQ facilitates secure, parallelized transfer of datasets between PowerScale solutions, providing strong benefits for media workflows. Replication policies can be set up or initiated ad hoc to transfer datasets between PowerScale solutions over the network.
- With SyncIQ, PowerScale is both a storage platform and a data transfer engine, so additional servers and transfer applications, which would incur additional cost and management overhead, don’t need to be implemented in front of PowerScale.
The PowerScale Backup and Recovery Guide provides more information about data movement capabilities in OneFS.
Quality of service
- Quality of service is increasingly important and has always been of interest for media workflows. SmartQoS is an embedded OneFS feature that monitors front-end protocol traffic over NFS, SMB, and S3. It allows limits to be set for the number of protocol operations to tie them back to performance SLAs, prioritization of workloads, and support for throttling to prevent specific clients from saturating a connection. I’ve seen an unthrottled copy job use the available connection bandwidth and interrupt a playout, so that’s an example of a use case where SmartQoS can be applied.
- Clients can be logically grouped and monitored with all kinds of metrics being captured to quantify and profile workloads. Introspection into read and write latencies, IOPS, and many other metrics on a per-protocol, path, IP address, user, and group basis can be captured and correlated in real time. Metrics can be tracked and enforced to provide quality of service for specific classes of users and workflows, which can all be defined to manage workloads.
For more information about quality of service in OneFS, see this blog post: OneFS SmartQoS.
Summary
The capabilities of PowerScale storage with OneFS are delivering unparalleled scale and feature benefits that elevate the capabilities of media entertainment use cases from the highest performance workflows to highly dense archives. Standardization on this enterprise-class, secure, and collaborative platform is the key to unlocking innovation and advancing your media pipelines.
1 Based on internal analysis of publicly available information sources, February 2023. CLM-0013892.
2 Based on Dell analysis comparing efficiency-related features: data reduction, storage capacity, data protection, hardware, space, lifecycle management efficiency, and ENERGY STAR certified configurations, June 2023. CLM-008608.
3 Based on Dell analysis comparing cyber-security software capabilities offered for Dell PowerScale vs. competitive products, September 2022.
4 Dell Technologies Executive Summary of Compliance with Media Industry Security Guidelines, https://www.delltechnologies.com/asset/en-ae/products/storage/briefs-summaries/tpn-executive-summary-compliance-statement.pdf.
Author: Brian Cipponeri, Global Solutions Architect
Dell Technologies – Unstructured Data Solutions