Content production has changed

Powered by technologies like Epic Games’ Unreal Engine, the creative landscape of Film, Episodic, & Live TV Broadcast has been changed forever.

With the advent of Virtual Production, which is essentially an extension of both Game Development and Post-Production practices, we have fundamentally changed the way that we create and interact with our content during production.

Due to this shift, many production facilities have been challenged by the exponential growth of content. Data continues to grow at an alarming scale, driven by increasing resolutions, expected quality of assets, and the sheer volume of content being produced to meet viewers’ insatiable demands.

Dell Technologies and Epic Games have come together to help solve this challenge, developing solutions that enable media companies to deliver massively scalable architectures that remain easy to manage, even at petabyte scale, and allow data to flow between datacenters, the edge, and the cloud.

How do we help?

This solution brings together one of the most trusted names in media storage – PowerScale OneFS – and one of the biggest names in Virtual Production technology – Unreal Engine.

 Figure 1. The partnership between PowerScale OneFS and Unreal Engine paves the way for new innovation in real-time production workflows

In the past, when working on levels / scenes in game engines like Unreal, all data required for playback had to be localised (copied to) to the machine being used to get the response times expected. This meant tasks required lengthy copies down to the workstation and lengthy saves back up to the storage when complete.

Faster ethernet technologies (10, 25, 50, 100GbE) are much more common in both workstations and access level switching. When you combine this shift in network throughput with PowerScale OneFS All-Flash nodes, Unreal editors can access their content simultaneously on PowerScale storage while also enabling modelers, riggers, texture artists, and so on to create and amend assets.

While all this live creativity is happening, on-set editors and colorists can perform ultra high-speed editing tasks on the footage to allow the Director of Photography to visualize the shot they want in real time. Fundamentally, this solution keeps actors acting, creatives creating, developers iterating, and directors directing - without having to worry about the constraints of the technology behind the shoot.

Figure 2. High-level infrastructure architecture diagram of data flow from datacenters to edge to cloud

Solution Benefits

Real-time edit

Enable more creatives to work simultaneously on more assets than ever before when using Unreal Engine with PowerScale All-Flash nodes, eliminating bottlenecks and driving creative productivity.

Build repositories

Build titles and levels quicker and more reliably using PowerScale All-Flash nodes as a lightning-fast centralized Artefact and Cache store.

Version control

Grow your production version control storage without causing downtime to your developers / editors by using PowerScale as the depot storage for your facility.

Unreal Engine

Unreal Engine 5 is the latest generation of Unreal Engine, a complete suite of creation tools for game development, architectural and automotive visualization, linear film and television content creation, broadcast and live event production, training and simulation, and other real-time applications. With UE5, game developers and creators across industries can realize next-generation real-time 3D content and experiences with greater freedom, fidelity, and flexibility than ever before.

Dell Technologies

At Dell Technologies, we believe great storytelling comes from creative people empowered by advanced technology. Our solutions for storytellers span the workflow, built with leading technologies across storage, servers, networking, workstations, displays, and partnerships with key media software and services vendors. These solutions provide media professionals with the foundation to continue driving innovation without technology getting in the way of the art, so they can deliver more powerful stories faster.

Ever increasing demand for content

Game development companies are facing increasingly difficult data challenges, accelerated by two primary drivers. On one hand, their data continues to grow at an unprecedented rate, driven by increasing fidelity, higher resolutions, and the sheer volume of content being produced to meet gamers’ insatiable demands. On the other hand, if that data were to become inaccessible to the development team and creatives working on game assets, your project would be held up due to fundamental capacity management issues. 

Scale without limits

This solution brings together the most trusted name in media storage – PowerScale (formerly Isilon) – and Perforce Helix Core, arguably the most trusted and widely used software version control system on the planet.

To help navigate change management and bring together the creative output of your teams, Perforce Helix Core gives teams of all sizes visibility and traceability into their projects and provides the Single Source of Truth that enables small teams to become more powerful and large teams to become more nimble. Helix Core offers stability to branching practices in distributed environments, full change traceability with secure asset versioning, and a scalable architecture suited to both artists and developers alike.

With your depots on PowerScale OneFS, you can scale their capacity nearly instantaneously without disruption to the running Helix Core application and without needing to schedule downtime of a studio.

Figure 1. Non-disruptive storage expansion

Additionally, utilizing technologies like NFS over RDMA and nconnect for NFS directly to PowerScale from Helix Core servers, we can deliver extreme single stream throughput speeds, ensuring that your Helix Core server is able to access versioned data in the fastest possible time.

Game development at scale

Combining Perforce Helix Core with PowerScale scale-out storage enables you to hit your deadlines and realize your creative visions with a joint solution catered to artists, developers, and system administrators alike. 

The joint solution of Helix Core and PowerScale accelerates workflows, supporting automated environments with tens of thousands of users running millions of daily transactions over petabytes of data.

Version large files of any type with Helix Core and Dell PowerScale, including code, config files, and large binaries. Developers can version and store everything – even in repos that can scale from hundreds of terabytes of data to petabytes in the same namespace – with no impact on system performance.

Feel secure knowing your successes can never outgrow the Helix Core and PowerScale storage system. Iterating fast, experimenting, and trying stuff out is just as much a priority to us as it is to you.

Additional benefits

PowerScale’s ‘Scale-Out’ architecture allows attached depots to grow seamlessly up to petabytes in a single namespace while remaining continuously online with no impact to servers or clients, bypassing the traditional constraints of having to upgrade an entire array or head, migrate LUNs, re-stripe data, concatenate LUNs, or expand dynamic volumes simply by allowing you to grow the entire filesystem on the fly.

Helix Core clients can also host their workspaces in NFS on PowerScale, which enables artists to work on their assets directly from the storage as if it were local without having to move data up and down between the server and their local machine’s workspace directory. Additionally, resources like build farms can access a shared NFS workspace so that only a single check-out is required to serve all builders.

Solution benefits

Scalable data management

Accelerate workflows. Nothing slows it down — automated environments, tens of thousands of users, millions of daily transactions, terabytes of data, etc.

Large files, all files

Version large files of any type, including code, config files, and large binaries. Developers can version everything — even in repos approaching hundreds of TB of data and 25 million files — with no impact on system performance.

Industry standard tool for artists

Give designers freedom to version art files quickly and easily from their preferred design tools while keeping all project files in a single source of truth.

One backlog that scales

Know your scope and prevent creep. Refine, prioritize, estimate, and give any other custom attribute to your deliverables.

Perforce Helix Core

Perforce Helix Core is the leading version control system for teams who need to accelerate innovation at scale. Store and track changes to all your digital assets, from source code to binary to IPs. Connect your teams and empower them to move faster and build better.

Dell Technologies

At Dell Technologies, we believe great storytelling comes from creative people empowered by advanced technology. Our solutions for storytellers span the workflow, built with leading technologies across storage, servers, networking, workstations, displays, and partnerships with key media software and services vendors. These solutions provide media professionals with the foundation to continue driving innovation without technology getting in the way of the art, so they can deliver more powerful stories faster.

Author: Andy Copeland, Global Solutions Architect - Media & Entertainment

Enable Avid Media Composer editors to work seamlessly on PowerScale OneFS

Capable of delivering fast access speeds alongside expansive capacities, PowerScale OneFS is at the heart of many production, post-production, and VFX workflows. It provides centralized collaboration storage between many editors across multiple disciplines and their content creation applications.

One of these content creation tools is Avid Media Composer. When working with Media Composer, bin locking provides an in-application method for editors to collaborate on their content. Using PowerScale OneFS and Projective Osiris together, Avid editors can easily collaborate on their projects in real time. The seamless integration of PowerScale and Projective Osiris enables post-production companies to extend their PowerScale investment to even more users and disciplines, providing the bin locking capability required for Avid workflows.

Projective

Projective Technology GmbH is dedicated to revolutionizing how content production happens. By focusing on the creative user and providing innovative technical solutions to production storage problems, Projective offers a fundamentally different approach to production storage and editing projects. Strawberry, Projective’s flagship Production Asset Management application, was created by editors for editors and is a recipient of the IBC Broadcast Engineering Award. Strawberry is used by broadcasters, production houses, ad agencies, film schools, and sports clubs worldwide.

Dell Technologies

At Dell Technologies, we believe great storytelling comes from creative people empowered by advanced technology. Our solutions for storytellers span the workflow, built with leading technologies across storage, servers, networking, workstations, displays, and partnerships with key media software and services vendors. These solutions provide media professionals with the foundation to continue driving innovation without technology getting in the way of the art, so they can deliver more powerful stories faster. 

Content production has changed…

Powered by technologies like Epic Games’ Unreal Engine, the creative landscape of Film, Episodic TV, and Live Broadcast has been changed forever. With the advent of Virtual Production and In Camera VFX, which are essentially an extension of both Game Development and Post Production, we have changed the way that we create and interact with our content during its production life cycle.

Because of this shift, many production facilities are challenged by the exponential growth of their content. Data continues to grow at an alarming scale, driven by increasing resolutions, the expected quality of assets, and the sheer volume of content being produced to meet viewers’ insatiable demands.

Dell Technologies and Epic Games Unreal Engine can work together to help solve this challenge, with solutions that allow media companies to deliver massively scalable architectures that remain easy to manage, even at petabyte scale, and allow data to flow between datacenters, the edge, and the cloud.

Solution benefits

Real-time edit

Enable more creatives to work simultaneously on more assets than ever before, when using Unreal Engine with PowerScale All-Flash nodes. Eliminating bottlenecks and driving creative productivity.

Build repositories

Build game titles and levels quicker and more reliably using PowerScale All-Flash nodes as a lightning-fast centralized artifact and cache store.

Version control

Using PowerScale as your depot and repository storage for your facility enables you to grow your version control storage without subjecting your developers and editors to the usual downtimes associated with disk expansion.

 How do we help?

This solution brings together a highly trusted name in media storage, PowerScale OneFS, with one of the most recognized names in Virtual Production and ICVFX technology, Unreal Engine.

In the past, when working on levels / scenes in tools like Unreal, all of the data for the game had to be localized to the machine you were working on in order to get the response times required. This meant lengthy copies down to the workstation to edit, and then lengthy saves back to the storage when done. Faster ethernet technologies (10, 25, 50, 100GbE, and beyond) are much more common in both workstations and access level switching now. When you combine this shift in network throughput with PowerScale OneFS All-Flash nodes, Unreal editors can access their content simultaneously on PowerScale storage, while modelers, riggers, texture artists, and others can create and amend assets on the fly as well.

While all of this live creativity is happening; on-set editors and colorists can perform ultra-high-speed editing tasks on the shot footage, ultimately giving the Director of Photography the freedom to visualize and capture the shot they want, in real time. Fundamentally, our solution keeps actors acting, creatives creating, developers iterating, and directors directing - without having to worry about the constraints of the technology behind the camera.

Figure 1.  A sample Dell Technologies infrastructure for Game Dev / Virtual Production / ICVFX

Unreal Engine

Unreal Engine 5 is the latest generation of Unreal Engine, a complete suite of creation tools for game development, architectural and automotive visualization, linear film and television content creation, broadcast and live event production, training and simulation, and other real-time applications.

With UE5, game developers and creators across industries can realize next-generation real-time 3D content and experiences with greater freedom, fidelity, and flexibility than ever before.

Dell Technologies

At Dell Technologies, we believe great storytelling comes from creative people empowered by advanced technology. Our solutions for storytellers span the workflow, built with leading technologies across storage, servers, networking, workstations, displays, and partnerships with key media software and services vendors. These solutions provide media professionals with the foundation to continue driving innovation without technology getting in the way of the art, so they can deliver more powerful stories faster.

Welcome to the first in a series of blog posts to reveal some helpful tips and tricks when supporting media production workflows on PowerScale OneFS.

OneFS has an incredible user-drivable toolset underneath the hood that can grant you access to data so valuable to your workflow that you'll wonder how you ever lived without it.

When working on productions in the past I’ve witnessed and had to troubleshoot many issues that arise in different parts of the pipeline. Often these are in the render part of the pipeline, which is what I’m going to focus on in this blog.

Render pipelines are normally fairly straightforward in their make-up, but they require everything to be just right to ensure that you don’t starve a cluster of resource, which, if your cluster is at the center of all of your production operations can cause a whole studio outage, causing impact to your creatives, revenue loss, and unnecessary delays in production.

Did you know that any command that is run on a OneFS cluster is an API call down to the OneFS API. This can be observed if you add the --debug flag to any command that you run on the CLI. As shown here, this displays the call information that was sent to gather the information requested, which is helpful if you're integrating your own administration tools into your pipeline.

# isi --debug statistics client list
        2023-06-22 10:24:41,086 DEBUG rest.py:80: >>>GET ['3', 'statistics', 'summary', 'client']
        2023-06-22 10:24:41,086 DEBUG rest.py:81:    args={'sort': 'operation_rate,in,out,time_avg,node,protocol,class,user.name,local_name,remote_name', 'degraded': 'False', 'timeout': '15'}
        body={}
        2023-06-22 10:24:41,212 DEBUG rest.py:106: <<<(200, {'content-type': 'application/json', 'allow': 'GET, HEAD', 'status': '200 Ok'}, b'n{\n"client" : [  ]\n}\n')

There are so many potential applications for OneFS API calls, from monitoring statistics on the cluster to using your own tools for creating shares, and so on. (We'll go deeper into the API in a future post!)

When we are facing production-stopping activities on a cluster, they're often caused by a rogue process outside the OneFS environment that is as yet unknown to us, which means we have to figure out what that process is and what it is doing.

In walks isi statistics.

By using the isi statistics command, we can very quickly see what is happening on a cluster at any given time. It can give us live reports on which user or connection is causing an issue, how much I/O they're generating as well as what their IP is, what protocol they’re connected using, and so on.

If the cluster is experiencing a sudden slowdown (during a render, for example), we can run a couple of simple statistics commands to show us what the cluster is doing and who's hitting it the hardest. Some examples of these commands are as follows:

isi statistics system --n=all --format=top

Displays all nodes’ real-time statistics in a *NIX “top” style format:

# isi statistics system --n=all --format=top
Node   CPU SMB FTP HTTP NFS HDFS  S3 Total NetIn NetOut DiskIn DiskOut
 All 33.7% 0.0 0.0  0.0 0.0   0.0 0.0   0.0 401.6  215.6     0.0     0.0
   1 33.7% 0.0 0.0  0.0 0.0   0.0 0.0   0.0 401.6  215.6     0.0     0.0

isi statistics client list --totalby=UserName --sort=Ops

This command displays all clients connected and shows their stats, including the UserName they are connected with. It places the users with the highest number of total Ops at the top so that you can track down the user or account that is hitting the storage the hardest.

# isi statistics client --totalby=UserName --sort=Ops
 Ops     In  Out  TimeAvg   Node  Proto  Class   UserName  LocalName  RemoteName
-----------------------------------------------------------------------------
12.8 12.6M 1.1k  95495.8     *       *      *      root          *           *
-----------------------------------------------------------------------------

isi statistics client --UserName=<username> --sort=Ops

This command goes a bit further and breaks down ALL of the Ops by type being requested by that user. If you know the protocol that the user you’re investigating is using we can also add the operator “--proto=<nfs/smb>” to the command too.

# isi statistics client --user-names=root --sort=Ops
 Ops     In   Out  TimeAvg   Node  Proto           Class  UserName        LocalName    RemoteName
----------------------------------------------------------------------------------------------
 5.8   6.1M 487.2 142450.6     1   smb2           write      root 192.168.134.101 192.168.134.1
 2.8 259.2 332.8    497.2      1   smb2      file_state      root 192.168.134.101 192.168.134.1
 2.6 985.6 549.8  10255.1      1   smb2          create      root 192.168.134.101 192.168.134.1
 2.6 275.0 570.6   3357.5      1   smb2  namespace_read      root 192.168.134.101 192.168.134.1
 0.4   85.6  28.0   3911.5      1   smb2 namespace_write      root 192.168.134.101 192.168.134.1
----------------------------------------------------------------------------------------------

The other useful command, particularly when troubleshooting ad hoc performance issues, is isi statistics heat.

isi statistics heat list --totalby=path --sort=Ops | head -12

This command shows the top 10 file paths that are being hit by the largest number of I/O operations.

# isi statistics heat list --totalby=path --sort=Ops | head -12
  Ops   Node  Event  Class Path
----------------------------------------------------------------------------------------------------
141.7     *       *      * /ifs/
127.8     *       *      * /ifs/.ifsvar
 86.3      *      *      * /ifs/.ifsvar/modules
 81.7      *      *      * SYSTEM (0x0)
 33.3      *      *      * /ifs/.ifsvar/modules/tardis
 28.6      *      *      * /ifs/.ifsvar/modules/tardis/gconfig
 28.3      *      *      * /ifs/.ifsvar/upgrade
 13.1      *      *      * /ifs/.ifsvar/upgrade/logs/UpgradeLog-1.db
 11.9      *      *      * /ifs/.ifsvar/modules/tardis/namespaces/healthcheck_schedules.sqlite
 10.5      *      *      * /ifs/.ifsvar/modules/cloud

Once you have all this information, you can now find the user or process (based on IP, UserName, and so on) and figure out what that user is doing and what's causing the render to fail or high I/O generation. In many situations, it will be an asset that is either sitting on a lower-performance tier of the cluster or, if you're using a front side render cache, an asset that is sitting outside of the pre-cached path, so the spindles in the cluster are taking the I/O hit.

For more tips and tricks that can help to save you valuable time, keep checking back. In the meantime, if you have any questions, please feel free to get in touch and I'll do my best to help!

Author: Andy Copeland
Media & Entertainment Solutions Architect

Here at Dell Technologies, we tested a cloud native real-time NDI ISO feed ingest workflow based on Gallery SIENNA, OneFS, and Adobe Premiere Pro, all running natively in Google Cloud.

TL; DR... it's awesome!

Mark Gilbert (CTO at Gallery SIENNA) had noticed there was a growing demand in the market for highly scalable, enterprise-grade file storage in the public cloud for ISO recording. So, we were excited to test this much-needed solution.

Sure, we could have just spun up a cloud-compute instance, created some SMB shares or NFS exports on it, and off you go. But then you quickly find that your ability to scale becomes an issue.

Arguably, the most critical part of any live broadcast is the bulk recording of ISO feeds, and as camera technology improves, recorded data is growing at an ever-increasing pace. Resolutions are increasing, frame rates are faster and internet connection pipes are getting larger.

This is where OneFS for Google Cloud steps in.

Remote production is now a must rather than a nice-to-have for every studio. The production world has had to adopt it, embrace it and buckle in for the ride. There are some great products out there to help businesses enable remote-production workflows. Gallery SIENNA is one of these products. It enables NDI-from-anywhere workflows that help to reduce utilization on over-contended connections.

You can purchase OneFS for Google Cloud through the Google Cloud Marketplace, attach it to a Gallery SIENNA Processing Engine via NFS export and start recording at the click of a button. In our testing, as soon as the recorders began writing, we were able to open and manipulate the files in Adobe Premiere Pro, which we connected to via SMB to prove out that multi-protocol worked too. This was all while the files were being recorded, and we could expand them in real-time in the timeline as they grow.

Infrastructure components (provisioned in Google Cloud):

• 1 x OneFS for Google Cloud
• 1 x Ubuntu VM
  • Running Gallery SIENNA ND Processing Engine
• 1 x Windows 10 VM
  • NDI Tools
  • Adobe Premiere Pro

We used a SIENNA ND Processing Engine to generate six real-time record feeds, three of which were 3840p60 NDI and the other three of 1080p30 DNxHD 145

One of the great benefits of using Gallery SIENNA ND on Google Cloud is that our ingest could have come from anywhere. We could have used any internet-connected device that can reach the Google Cloud instance, be that a static connection in a purpose-built facility or a 4G/5G cell phone camera on the street with the NDI tools on it.

High-level workflow:

1. Added a Signal Generator node (3840p60) into our SIENNA ND Processing Engine instance
2. Used the SIENNA ND node-based architecture to add on a timecode burn and frame sync
3. Added 3 x <NDI Recorder>
4. Configured the recorders to write out to an NFS export on our OneFS for Google Cloud instance
5. Added a StreamLink Test node (1080p30) into the same SIENNA ND Processing Engine instance
6. Added timecode burn and frame sync nodes again
7. Added 3 x <DNxHD 145 Recorder>
8. Configured the recorders to write out to the same NFS export on our OneFS for Google Cloud instance
9. Hit record on all recorders

Once the record was running, we added a "Media Picker" node and selected one of the files that we were recording. Then, we connected this growing file and one of the frame-sync outputs to a "multiviewer" node. We then watched both the live feed and chase play from disk as it was being laid down.

To cap it off, we also mounted one of the output paths using SMB from a Google Cloud hosted Windows 10 instance running Adobe Premiere Pro, and we were able to import, scrub and expand the files as they grew in real-time, allowing us to chase edit.

To find out more about the Dell Technologies offers for Media and Entertainment, feel free to get in touch by DM, or click here to find one of our experts in your time zone.

See the following links for more information about OneFS for Google Cloud and Gallery SIENNA.

Author: Andy Copeland