Today’s Challenges for Post Production, Visual Effects and Playout Workflows
Today’s broadcast and post production environments are faced with a multitude of challenges, from complex and difficult to integrate ecosystems to increased pixel depth/frame rates to the evolving IP-video landscape. Chief engineers and IT administrators in many cases must integrate MAM, VFX, grading, transcoding, rendering/compositing workflows with make do cobbled together systems from disparate vendors, often resulting in expensive, unwieldy and nearly intractable piles of hardware and software. For instance, your MAM speaks NFS, your object store uses S3, your color artists all have Windows, and your render farm just isn’t fast enough. You bought storage for each one and you’re pondering how to integrate them all with limited staff. Adding to the debilitating stress level are the need for regional, national and global collaboration and fast turnaround project deadlines while keeping capex and opex costs in-line. All these facets of post-production to on-air and distribution workflows present one very challenging storage environment for companies in the M&E landscape.
What is needed to address this growing dilemma of not having to rely on monolithic or cobbled together hardware and software is a scalable, cost-effective, hardware agnostic storage software solution that delivers the IOPS, throughput, and resiliency to tame even the most formidable workflow needs. Further, it should be a system designed to ease a facility’s administrative burden, prioritize cost containment, provide seamless differing file system support, and afford broad connectivity choices in today’s highly responsive containerized world vs. previous years’ inflexible data center designs.
The Quobyte Solution for Post Production, Visual Effects and Playout Workflows
Enter Quobyte (www.quobyte.com), a Silicon Valley headquartered company founded by ex-Googlers Dr. Björn Kolbeck (CEO and Co-Founder) and Dr. Felix Hupfeld (CTO and Co-Founder). Quobyte’s unique and innovative parallel distributed file system with Google-style operation and functionality provides a single unified storage platform and namespace for the variety of file types in the workflow and gives post-production teams fast access to their files. Each person in the workflow, whether on a Mac®, Windows®, or Linux machine can access the same file. These same files can also be accessed via the company’s stateless S3 proxy for collaboration among everyone involved in the post-production process. The software supports parallel, sequential and small block random I/O workloads, and all data is available on any server or virtual machine. Quobyte thus eliminates silos of content and delivers a unified storage experience for productive collaboration.
From a 10,000 feet perspective, the benefits to a post production to VFX to playout workflows are numerous.
Post Production Workflow Benefits:
- Increased collaboration and productivity
- Eliminates needlessly copying, exporting, and uploading files between systems and platforms
- More efficient use of network bandwidth and storage capacity
- Eliminate management overhead and complexity
- Automatic file prefetching for jitter-free playback of 4K streams from applications such as NUKE or Autodesk’s Flame
Visual Effects Workflow Benefits:
For Visual Effects (VFX), Quobyte’s software storage solution provides high IOPS with consistent sub-millisecond latency to deliver the raw performance compositing and rendering demand. The same Quobyte system also delivers parallel I/O for unrivaled throughput to support concurrent streams of HD, 2K and 4K video with jitter-free playback. Scaling Quobyte storage for increased performance and capacity is easy by simply adding additional nodes as needed, even on the fly. Reliability and data safety are built-in, no matter how large a facility scales its environment, resulting in constant data availability for VFX workflows. The benefits are numerous, including:
- High-performance shared storage to support all aspects of a media and entertainment workflow
- A single storage platform eliminates data silos and promotes increased collaboration
- Easily scale storage performance and capacity
- Built-in reliability and data safety so data is always available
- No downtime for maintenance or when adding capacity or bandwidth
- Easy automated administration
Playout, Streaming and CDN Benefits:
Quobyte’s software storage provides a single platform and namespace for storing video in any format for playout, streaming and CDN origin storage. Its complete fault tolerance and automation capabilities ensure 24-7 operations with no need for human intervention even when facing hardware outages or running software updates and system upgrades.
A CDN needs an origin storage system that becomes a critical performance component on cache miss or application request. Quobyte’s software solution provides the near-instantaneous high throughput and scalable bandwidth needed to reduce loading times to reduce request-to-playout lag times.
By using commodity HDDs and SSDs, broadcasters, VOD streaming and OTT service providers can provide cost-efficient, scalable capacity and performance when it’s needed and where it’s needed. Quobyte for Playout benefits are:
- Software storage for a flexible and responsive infrastructure
- Built-in fault tolerance for “always-on” operation
- Cost effective, multi-petabyte content stores built on commodity hardware (HDD, SSD, NVMe, SMR)
- Easily scale storage performance and capacity
- Eliminate storage management overhead and complexity
- Support for Mac, Windows, Linux, and S3
Quobyte’s Unified Storage Access
Quobyte’s system uses a Linux FUSE (Filesystem in Userspace) client for a facility’s render farm and gets high IOPS/low latency and parallel I/O performance for up to thousands of nodes. Camera files can be ingested via NFS or SMB. Its unified storage access lets Mac or Windows workstations work on those very same files. No migration, no duplication, no data movement required. And when done, users can transparently migrate older files to an erasure coded volume to save space. Users can even serve them up via a stateless S3 proxy. Moreover, Quobyte’s storage software allows users to specify (down to the individual drive) exactly which hardware is accessible, and by whom if projects need to be isolated for legal reasons. Full support for multi-tenancy gives administrators total control, while on-the-fly reconfigurability means changes take only minutes, with zero down time.
Automated Management of Resources
Quobyte’s low-touch solution also frees administrators from worrying about system internals. A JSON-RPC API supports integration with existing automation tools. Users can reconfigure their entire storage with one click; bring agile best practices to their storage. The software is self-managing, shielding users against hardware failures, operator errors, network partitions and disk corruption. End-to-end checksums protect all file data, metadata, and communications, eliminating the need for hardware and software RAID controllers.
Aside from enterprise-grade provisioning, policy-based resource allocation and integrated monitoring and analytics, Quobyte’s solution also features built-in hardware management. Users benefit from automatic scrubbing, detection, and repair of corrupted data, and the software can drain a server with one click. Also built-in are monitoring and hardware watchdogs that can detect and initialize new devices and perform device lifecycle management for “lights-out” operations.
Enterprise Data Center Ready, Highly Configurable
Quobyte’s software storage solution is aimed at operational ease and flexibility. Users are free to deploy on any hardware, and cluster nodes don’t have to be identical. Features like user-definable data placement, policy-driven tiering rules, and data pinning give storage administrators the freedom to optimize for their environment with no vendor lock-in. And with full support of all modern Linux distributions with no kernel dependencies, the software is extremely flexible. Native Linux, Windows, and Mac drivers deliver high performance parallel I/O. NFS or S3 workloads can be connected through stateless proxies and storage is accessible through all interfaces simultaneously. Quobyte’s software also allows facilities multi-thousand node scalability from four nodes to entire data centers and beyond, affording additional nodes to be added to increase performance without downtime. In addition, Quobyte’s software solution offers unlimited data capacity and non-disruptive live updates.
Fast and Reliable Data Access with Full Fault Tolerance
Delivering hyper-performance storage while maintaining data availability despite disk or network failures takes a holistic design approach. Quobyte took ideas from high-performance computing technology and 10 to 15 years of research on fault tolerance with distributed systems, based on the Paxos algorithm. The result is a software solution purpose built to handle real- world environments, even node failures while keeping its operational integrity intact. True end-to-end checksums of every data block assures data integrity. Ultra-low latency and fast metadata operations keep up with the toughest small-file workloads. Hardware monitoring will offline an errant disk before it causes corruption, and self- healing heuristics route around node failures. The result – no single point of failure, partition tolerance and fully automated fault tolerance on commodity hardware.
Future Proof for Today’s and Tomorrow’s Workflows
Yesterday’s inflexible data center designs are quickly giving way to tomorrow’s highly responsive containerized world. Quobyte’s software provides persistent container storage and it can also be run inside of the containers. It delivers low-latency and parallel throughput performance to all containerized applications, so users can run databases, scale-out applications, and big data analytics on one single infrastructure.
The software works out of the box with a facility’s technology stack; combining it with Kubernetes, Mesosphere, or Docker allows users to set up an ultra-responsive and reliable storage infrastructure with ease to address current and future rapidly fluctuating workloads. Quobyte also supports OpenStack’s Cinder, Glance, Manila, and S3 interfaces, and is also compatible with Hadoop – resulting in a seamless, unified storage access architecture.
For broadcasters, studios and post production facilities facing continuing budget constraints and manpower reductions, the Quobyte solution offers a flexible, unified and collaborative storage experience that helps users cost-effectively scale out their storage performance and capacity requirements when needed, dramatically reduce administrative overhead while providing guaranteed and secure data integrity for the entire workflow chain.
His agencies’ services, for over a quarter century, have helped garner early stage and growth companies significant brand exposure, resulting in either acquisition or IPO, with a total valuation of over $2.5 Billion.
Mr. Chan is a serial entrepreneur, philanthropist, author, mentor and angel investor. He played key executive roles early on in helping to pioneer and usher in both professional digital audio and digital video technologies for the recording, broadcast, cinematic and post production industries. He also played key executive roles in helping to develop and market data storage technologies. His expansive background spans four decades in ever increasing executive roles in operations, business development, engineering and sales/marketing in the information technology, media & entertainment, storage and networking, and other related high technology industries.
He is an active mentor and senior advisor to many established and VC backed start-up companies and incubators, a market/technology analyst, and has over three decades of experience in brand development, management consulting, company turnarounds, creative advertising and public relations.
A graduate of California Polytechnic State University (Cal Poly), San Luis Obispo; Mr. Chan recently ended his 14th year as Vice President for the Fullerton College Foundation; 22nd year as an Entrepreneur-in-Residence and mentor for the Small Business Institute at California State University Fullerton and incubators. Additionally, he is a volunteer member of DECA (Delta Epsilon Chi and Distributive Education Clubs of America); an advisor to the College of Engineering's Global Waste Research Institute, Cal Poly, San Luis Obispo and a member of the President’s Green and Gold Society; and advisory council member to Pepperdine University's Most Fundable Companies program. Passionate about hospice care, he was the past Board President for the Healing Hearts Association; and is a regularly featured speaker at many MBA and Doctorate level lectures around the country. He has presented and published over 30 papers worldwide, is a book co-author, an Editorial Board Member for the Society of Motion Picture and Television Engineers, past Senior Editor for Broadcast Beat and AV Beat Magazines, Computer Technology Review, and is a regularly featured Contributing Editor to many US and international trade publications for over 35 years.
View Cognitive Impact services at www.cognitiveimpact.com. Curtis can be reached at [email protected]; Office: (714) 620-4993
- Vivaro Media Selected as the Official Broadcast Distribution Partnerfor the CIC Mont Ventoux Professional Cycling Race - May 30, 2023
- Axle AI launches AI-powered MAM for OpenDrives storage platform at NAB - April 18, 2023
- Matrox Blueprints Next-Generation On-Premises and Cloud-Based Live Production Workflows at IBC 2022 - September 9, 2022
