Tom Coughlin, Coughlin Associates
The Covid-19 pandemic has driven many post production organizations to remote work. This has caused shifts in the way media content is being stored. One important element is a greater reliance on cloud storage, whether from a private data center or through the hyperscale cloud storage suppliers. In this article we will look at our projections for the overall growth of post-production storage and then are offerings and insights from the 2020 IBC, the 2020 NAB Show New York and various company briefing that will help post-production facilities to continue operation, control their costs and increase their productivity.
The figure below plots the annual demand in storage capacity for post-production, including NLE, breaking out direct attached and network attached post-production storage capacity[1]. We include a separate breakout of remote (cloud) storage for collaborative workflows. Note that due to the Covid-19 pandemic and many people working at home through most of 2020 and likely part of 2021 we project a significant bump in the use of cloud storage for post-production starting in 2020 compared to 2019 (from 8% to 20% respectively) and continuing to increase through 2025.
With the growth in post-production cloud storage we will first look at developments that incorporate cloud storage in media and entertainment workflows from various vendors.
Cloud Storage for Remote Post Production
Avid’s Nexis 2020 storage solution provides collaborative workflow from anywhere on rich media. It also provides 40% more shared storage in the same footprint using higher capacity HDDs, optimizes content mirroring to eliminate downtime and data loss and flexible storage tiering that combines on-premises and cloud resources. It also provides broader support for third party tools.
Interest in using the cloud in post-production workflows is growing. Avid did a survey of their customers in 2020 before the Covid-19 pandemic hit and found that only 20% of customers don’t plan to use cloud storage, while 40% said they would use less than 100TB, 30% said that they would use 0.5-1PB and 10% said they would use more than 1PB of cloud storage. The figure below shows Avid’s lineup of storage products including Avid Nexis/Cloudspaces to integrating on-premises to cloud storage.
Avid also said that it has done a soft launch of running Avid Media Composer on a virtual machine and using Kubernetes containers in an editing environment, accessed using Teredici, with Nexis cloud storage providing edit on-demand as shown below.
Scale Logic was showing there Remote Access Portal, a 1U Linux appliance that enables remote access to on-premises storage for both proxy and high resolution workflows, as shown below. Note that a remote client local HDD or SSD can be used as local cache to improve performance.
Synchonization processes happen in the background without an editor having to do anything and a project saved remotely syncs back to the on-premises storage automatically so others can see the changes.
Editshare was awarded the NAB Show product of the year award at the 2020 NAB NY conference. The company released a new version of its EFS 2020 file system in July 2020. According to the company, “The media-optimized file system features security improvements at every layer and enhanced performance across the board. In addition to the powerful storage management tools built into EFS, the new RESTful API opens the door for customers and technology partners to automate advanced storage management workflows in a secure environment. Fully compatible with the latest version of FLOW, EFS enables media organizations to build extensive collaborative workflows, shielding creative personnel from the underlying technical complexity while equipping technical teams with a comprehensive set of media management tools.”
The latest version of EFS supports cloud workflows including AWS, Tencent Cloud and others. IT managers and administrators have fine grained control over content, folder structures and content flow to enable better collaboration across multi-site and multi-project operations.
EditShare also said that it has been helping broadcasters and media companies increase their remote content production output with its EFS shared storage and FLOW media management solution. The company said that during the pandemic the Philippine Long Distance Telephone company (PLDT) implemented the comnpany’s collaborative solutions to automate more than 50 manual workflows and increase remote production content output by up to 40%. The image below shows remote post production using EditShare products
Facilis was at the 2020 virtual NAB show NY. Facilis provides high performance shared storage for collaborative media production. The company’s recent developments include version 8.05 of the Facilis Shared Storage Systems, version 3.6 of its FasTracker Media Asset Management software and the new Facilis Edge Sync for remote access for display.
The Facilis Shared Storage Version 8.05 includes software defined bandwidth priority, SSD tiering and multi-disk parity. Bandwidth Priority delivers full throughput to all workstations during normal operation but prioritizes workstations to maintain greater throughput when the server enters a high-load condition. This priority setting is dynamic and can affect client performance within seconds of applying.
Software-defined Multi-disk Parity can be enabled for up to 4 drive failures per drive group, on a project-based, virtual volume-basis. This technology allows owners of aging systems to better protect their assets from data loss due to drive failure. SSD and HDD tiering were developed to deliver dedicated speed for projects needing SSD-level performance, while maintaining a perpetual HDD-based mirror.
Facilis Edge Sync starts with Facilis Object Cloud software which utilizes a native Facilis virtual volume as the local disk cache and adds a dedicated Azure Cosmos DB database to synchronize multiple desktops together into a single file system. With the Facilis Edge Node installed in remote locations, the path of the media files and project files are exactly the same, whether you’re working in-facility or at home. Any changes or additions to the project files is instantly updated in every location. The figure below
Cinesite partnered with Qumulo and AWS to let its animation and VFX pipelines leverage Qumulo’s hybrid file data services to deliver up to 16K rendered video. When faced with issues of intermittent freezes with a recently purchased storage cluster Cinesite approach Qumulo, who quickly deployed hardware nodes onsite and got the company running again.
Later, in order to burst to the cloud to scale its rendering capability, the company impolemented Qumulo cloud storage which allowed the organization to spin up machines and store data on AWS. A Qumulo case study says that “Qumulo’s hybrid file software runs the same enterprise file system in the cloud as on-prem, and data can be natively and seamlessly replicated between instances or across regions. Bursting to 20, 200, or even 2,000 high-quality render nodes on AWS with Qumulo to keep pace with all that power is no problem. Instances can be spun up in minutes, and torn down just as quickly. “
Integrated Media Technologies (IMT) announced that its SoDA software has been integrated with Dalet’s Ooyala Flex Media Platform to simplify the secure transfer of artificial intelligence and machine learning worlflow content to the cloud. The integration of IMT SoDA software and Dalet’s Ooyala Flex Media Platform is offered in the United States and Europe.
The joint solution will address media asset data management requirements by streamlining the transfer of large data workflows while delivering a simplified data movement approach for creative content. Post production and media services customers will be able to predict the cost and time to move files using SoDA before a storage transfer, enabling them to make smart decisions on data management and to keep project costs at or below budget.
Increased reliance on the cloud with remote collaborative work will grow at a higher rate after the current pandemic experience has ended. Whether in the cloud or on premises, various solid-state storage solutions will help editors deal with the increasing size of video content requiring higher performance storage to provide the real time experience that these professionals require. Let’s look at some of the latest solid state storage solutions for the M&E industry.
Solid State Storage Solutions
NetApp’s Cloud Manager provides policy driven management of application storage and data across multiple public cloud providers and on-premises locations. The Spot by NetApp Product Suite providing cloud infrastructure analytics, cost optimization, capacity optimization and workload optimization for Kubernetes containers. The company’s ONTAP 9.8 provides increased cloud integration and data availability for enterprise applications. ONTAP 9.8 provides a hybrid cloud cache architecture, continuous availability and unified data management over SAN, NAS and object storage.
NetApp has long supported Dreamworks animation, which requires a balance of storage capacity and performance. The new FAS500f (shown below) is an all flash capacity oriented storage array (up to 734TB raw capacity with an expansion shelf) using QLC flash SSDs to provide higher capacity. This product has end to end NVMe support and is managed by NetApp’s ONTAP software. The product is targeted at high volume unstructured data applications, such as media and entertainment and animation.
At the 2020 IBC ATTO was showing their SiliconDisk RAM-based, high performance storage appliance, with advertised capacities of 128GB and 512GB. Using RAM, rather than flash memory, this product provides much higher performance, for a price.
This product provides latencies less than 600 ns and up to 6.4M 4K IOPS and data transfer bandwidth up to 25 GB/s. It comes with 4 100 Gb Ethernet ports for a total of 400Gb of bandwidth. According to the company, “Data is instantly stored and retrieved with incredible speed letting you edit more streams of video, capture more data instances for AI/ML, manipulate more data sets quicker, and provides incredible performance to index look-ups.”
The SiliconDisk includes a real time optimizer that provides performance analytics on your storage network connections, storage utilization and overall SiliconDisk data performance. It also has an xCORE I/O acceleration, handling reads and writes with almost zero additional processing overhead. Also, using DRAM, rather than flash memory, the system doesn’t need to manage media wear.
Excelero announced that DigitalFilm Tree used its elastic NVMe storage (NVMesh) to provide 10X faster rendering processing and 100X faster storage. According to the company, “NVMesh’s software-defined distributed block storage for high performance computing workloads empowers users through better storage. Customers benefit from shared NVMe resources across the network, access to remove NVMe at local speed – and performance that exceeds the capacity limit of local flash on servers.
In the NVMesh the data-path runs exclusively on the client side, involving no CPU cycles on the server side. This is particularly interesting for hyperscale applications as there is no noisy neighbors effect. An important component in the NVMesh architecture is the TOpology MAnager (TOMA), an intelligent cluster management component that provides volume control plane functionality and enables data services such as RAID, erasure coding and data sharing (among client machines). The direct effective data path from the application to the NVMe storage is illustrated in the image below.
As an example of NVMesh at work, the team producing Prime Rewind: Inside The Boys, a before show for season 2 of Amazon Prime Video’s superhero and vigilante series The Boys, DFT’s system faced a test to its new Excelero-powered storage solution. The production team needed to process 40 hours of client-uploaded dailies, back them up, make proxies for rapid editing, process and deliver them to their editorial department – in just 10 hours.
VAST offers what it called its Data Universal Storage that uses Intel Optane NVMe SSD as a cache layer for a QLC NVMe storage layer as shown below. The company says this storage architecture provides high performance lower cost storage that is being used in animation studios, sports leagues and broadcast.
At the start of the IBC, Cloudian announced that its HyperStore object storage software is now flash-optimized, enabling enterprises to meet the needs of performance-intensive workloads while deploying flash and HDD-based nodes with an adaptive hybrid architecture allows customers to reduce total cost by 40% by tiering less frequently used data to the HDD storage. HyperStore is available either as a software-only solution or in a pre-configured appliance, the HyperStore Flash 1000 series. The HyperStore Flash 1000 offers 77TB and 154TB capacities in a 1U form facto and is shown below.
According to the company, “Cloudian’s new flash-optimized software delivers the needed performance while providing all the advantages of Cloudian’s enterprise-grade object storage platform, including fully native S3 compatibility, industry-leading security and advanced management features such as multi-tenancy and quality-of-service. Flash-optimized HyperStore leverages the reduced-latency I/O profile of flash media on industry-standard hardware, delivering partial-object reads and low latency data accesses at scale. Cloudian’s platform is certified with leading NVMe suppliers such as Intel and Kioxia and is Intel Optane-ready for even greater performance.”
Open Drives announced the availability of its Atlas 2.1 software platform that powers its OpenDrives storage solutions. The new software operates on the company’s recently released Ultra Hardware platform, shown below, which now includes NVMe SSDs in its Ultimate product and balanced with HDDs in its Optimum product. These are shown below as well as the Momentum HDD array product.
Atlas 2.1 has features allowing companies to massively scale-out while maintaining scale-up performance. These features include: storage clustering, distributed file systems, containerization, conditional automation, centralized management and visibility, cloud storage support, and high-availability.
Storage Clustering allows individual scale-up devices, or nodes, to be aggregated together forming a cluster. This parallel distributed architecture enables balanced workloads among cluster nodes without sacrificing performance hits such as increased latency.
Containerization brings functions such as compute and the application itself closer to where the data resides in storage. OpenDrives approached containerization from the storage point of view. Through this, the company says that OpenDrives is able to achieve huge performance gains by intelligently and efficiently delivering the data to the container.
Conditional automation is a complementary feature to containerization, enabling triggering actions, such as time-based or file-based actions, to create automated tasks that fire up and operate independently of other tasks. Centralized management and visibility through a single pane of glass gives operators insights into storage infrastructure efficiency and helps to better configure settings to tune nodes and storage clusters.
Cloud storage support enables users to send and receive both on-premises and cloud data via the S3 protocol and sharing S3 remote targets through Service Message Blocks (SMB) locally. High availability manages operational continuity so customers can configure standby nodes that activate when a primary device goes down.
The cloud and solid-state storage are changing the way we work on media content. But for professionals working at home or in a small facility local storage can provide the highest performance. Let’s look at new and updated local storage offerings for M&E applications.
Local Workflow Storage Products
Promise Technology introduced its PegasusPro, at the 2020 IBC, a Thunderbolt 3 DAS and NAS fusion system intended to improve workflow efficiency in digital media collaboration. The product provides rapid data transfer from DAS to 10GbE NAS and vise-versa using the company’s FileBoost technology. The company says that multiple people can connect directly to PegasusPro via Thunderbolt 3 and simultaneously share their work with other team contributors over the NAS. The PegasusPro product line is shown below.
Seagate is offering its EXOS HDD JBODs as well as its Nytro All flash arrays.
One Stop Systems had a virtual booth tour at the 2020 IBC. The company says that it, “designs and manufactures high performance specialized systems for the media, entertainment and visualization industries, utilizing the power of PCI Express, the latest GPU accelerators and NVMe storage to run demanding compute applications including final frame rendering, large scale event visualization, real time extended reality and AI enhanced video post-production. OSS offerings include the industry’s first PCIe Gen 4 based render accelerators and video recorders with twice the bandwidth of existing systems and up to 16 NVIDIA A100 GPUs in a single system. OSS delivers AI on the Fly™ bringing datacenter performance to on-location and in-studio workflows.”
Synology announced its DS1621xs+ at the 2020 IBC. According to the company, “the DS1621xs+ shares the powerful Xeon processor found in other Synology data center devices. Over 3.1 GB/s seq. read and 1.8 GB/s seq. write performance means it can tackle larger data sets and handle more users, at exceptionally fast speeds. It’s also paired with ECC memory for maximum dependability, and when combined with Btrfs and other comprehensive data backup options, users can be confident their data is safe.” The product is shown below.
Six internal 3.5” HDD bays enable up to 96TB of raw storage capacity. Expansion units allow increasing this to 16 bays and 256TB capacity. An additional 10GbE NIC can speed up rendering projects or provide even faster transfers for multiple virtual machines. The product provides a local NAS service, while allowing remote access with a web browser or mobile app.
Symply introduced its upgraded SymplyWORKSPACE for the 2020 virtual IBC, a StorNext6 powered desktop multi-user Thunderbolt 3 SAN storage system with embedded axle ai 2020 AI-based media asset management with storage capacities from 48 TB up to 366 TB, shown below. The company says this unit can support up to 8 simultaneous users collaborating on 4K jobs and supports remote access, so users can access proxies on their system from home. Adv aced RAID protection helps protect media assets and the use of solid state storage helps improve overall system performance. The axle ai runs in a Linux virtual machine on the WORKSPACE Xeon processor.
The 2020 Digital Storage in Media and Entertainment Report
The 2020 Digital Storage for Media and Entertainment Report, from Coughlin Associates, provides 251 pages of in-depth analysis of the role of digital storage in all aspects of professional media and entertainment. Projections are given out to 2025 for digital storage demand in content capture, post-production, content distribution and content archiving are provided in 62 tables and 129 figures.
The report benefited from input from many experts in the industry including end users and storage suppliers, which along with economic analysis and industry publications and announcements, was used to create the data including in the report. As a result of changes in the economics of storage devices higher performance solid-state storage will play a bigger role in the future. The cloud and hybrid storage including the cloud have assumed a new importance for many workflows during the Covid-19 pandemic. When the pandemic passes, use of cloud storage will continue to grow in the media and entertainment storage market going forward.
You can find out more and order direct at tomcoughlin.com/product/digital-storage-for-media-and-entertainment-report/
[1] 2020 Digital Storage in Media and Entertainment, Coughlin Associates, tomcoughlin.com/product/digital-storage-for-media-and-entertainment-report/
About the author
Tom Coughlin, President, Coughlin Associates is a digital storage analyst and business and technology consultant. He has over 39 years in the data storage industry with engineering and management positions at several companies. Coughlin Associates consults, publishes books and market and technology reports and puts on digital storage-oriented events. He is a regular storage and memory contributor for forbes.com and M&E organization websites. He is an IEEE Fellow, Past-President of IEEE-USA and is active with SNIA and SMPTE. For more information on Tom Coughlin and his publications and activities go to www.tomcoughlin.com.
- Spatial Mic Converter Plugin Updated To Include New Visualization Modes - June 2, 2023
- Sencore Welcomes Chad Hassler as New Vice President of Sales - May 31, 2023
- NAB Show Announces Winners of NABiQ - May 24, 2023
