Multi-Cloud Data Management with SwiftStack

Earlier this month, I wrote an article on Object Cloud Storage and featured Caringo, a pioneer in the marketplace. There are a great many players vying for the M&E market due to its forecasted data growth in the coming decade. The competitive landscape is fierce, and I can see consolidation coming in the years to come, but for the moment, vendors can be broken down into three categories. At the top of the heap are large established incumbent storage hardware vendors who have sold the prior-generation of storage before cloud storage such as EMC, NetApp, IBM, HP, HDS, Dell, HGST and Oracle, followed by public cloud providers like Amazon Web Services (AWS), Google Cloud Platform (GCP) and Microsoft Azure now courting studios and datacenters to move their workloads into the cloud. Mid-level players include, but not limited to Data Direct Networks, Quantum, and Panasas. Then there are companies with solid histories that have pivoted their business model from pre-cloud to cloud including Scality, Cloudian, Caringo and SwiftStack. Lastly, early stage startups such as Minio, OpenIO, Tarmin and others that have had the opportunity to see the missteps of others, promises to bring a new level of sophistication and ease-of-use to the industry.

With such a wide variety of vendors and solution choices in-play, one could argue that there may be too many proprietary object storage solutions for on-premises to choose from, all of which were not architected to natively support public cloud storage as part of a single namespace. It’s to this point that this secondary article features multi-cloud data management company SwiftStack, another major vendor in the space. SwiftStack is unique in that it was designed for the cloud, and is one of the few vendors to offer consumption-based pricing organizations are looking for. Moreover, other solutions support freedom of choice for hardware, though SwiftStack provides the most flexibility with choice of server and choice among several Linux distributions, including Red Hat, CentOS and Ubuntu.

Recently, Mario Blandini, Vice President of Marketing for SwiftStack answered some questions for me and the following are the highlights.

The Beginning of SwiftStack and its Eureka Moment
Bay area based SwiftStack which began operations in 2011, is a major player in the hybrid cloud storage space for enterprise level companies. The company was the vision of founder Joe Arnold, currently its President and Chief Product Officer. Arnold brings years of cloud expertise from roles at Yahoo, as well as executive positions in cloud technology startups. Anders Tjernlund is the co-founder and is COO of SwiftStack, having spent years at HP, as well as several startups focused on cloud and open source. The third person responsible for SwiftStack’s continuing rise is Don Jaworski, which began with SwiftStack as a member of the Board of Directors, and joined full time as the CEO in January of 2015. Prior to SwiftStack, Don advised several startups in addition to having top executive roles at industry incumbents NetApp, Nokia, Brocade, and Sun Microsystems. To date, SwiftStack has raised $26.4 milion from venture investors, operating as a Series B startup with greater than 100 customers fueling company growth.

The Eureka moment did not occur with media production workflows in mind, instead came while Arnold was working in Korea building the application storage architecture for Korea Telecom. He realized that even the most sophisticated organizations would struggle in moving classic applications to cloud infrastructure, and saw the opportunity for a product solution that could offer customers a similar level of service to Amazon or Google, but with the flexibility to store data across clouds. SwiftStack was first used by Web / SaaS companies, as well as large online gaming companies who built their own storage clouds using SwiftStack’s technology.

Eliminating Growing Pains with SwiftStack
According to Blandini, SwiftStack’s technology is purpose built to resolve many of the challenges facing organizations. Common to most of the vendors’ solutions are a reduction in datacenter footprint, AC/Power savings, malware security, collaboration efficiency, multi-file support, file parsing for transcoding/rendering efficiency and third- party application transparency, to name a few. In addition, SwiftStack differentiates itself in IT datacenters with the following:

Enterprise Agility – Applications operate without the need for IT action so business units have faster response to changes and opportunities.

Elastic Scale – Resources adapt to business needs across regions without limits, supporting the transition to cloud and avoiding cloud lock-in.

Business Innovation – New architecture and cost savings unlock new opportunities for development and differentiation up the stack.

Future Proofing Metadata – SwiftStack’s technology provides universal access to unstructured data, policy automation for data placement across clouds and more, and a metadata platform to maximize the value of data. Metadata is the organizing principle for data management in the cloud. The power of SwiftStack enables new applications that have yet to be conceived, where metadata makes future value from data possible. Noted Blandini, “developers use an SDK to leverage new ways to discover, visualize, analyze, enrich, and transform massive amounts of data. Applications transform stored data for serving users on-demand to keep up with mobile consumption and presentation.”

Further, SwiftStack is differentiated from virtually all private cloud storage systems across use cases with its pay-as-you-grow licensing model, allowing service providers and enterprises to conserve capital expense and deliver against customer requirements at the lowest possible price over time, enabling those customers to compete with the largest public cloud providers.

SwiftStack is also differentiated in freedom of choice among hardware across use cases, with an ability to drive differentiated entry configurations for backup that scale-up from the starting nucleus without needing to re-architect the cluster. Particularly for workloads in data-centric environments like M&E and Life Sciences, having storage that supports both the apps of today, as well as the apps of the future is differentiated, where any data written by any access method is accessible via file services or object APIs.

“SwiftStack believes that object storage itself is not the market where our product will be most successful, but going forward, the ability to manage data across multiple clouds in native cloud format is the solution that our product best delivers,” noted Blandini. Being software-only, SwiftStack acknowledges the challenge for users to install and configure SwiftStack compared to buying a ready-made appliance. This is because SwiftStack was designed from the ground-up to be software that’s installed on top of standard Linux running on standard servers. Ease-of-use was a focus since their inception, and having a fully-functional download helps overcome the objection, where the company invites users to see for themselves that software-only can work. “Many are amazed how fast they were able to set up a distributed storage cloud or how simple it was to add additional capacity or performance.”

SwiftStack’s Underlying Technology
Rather than design the core platform 100% from scratch, the company chose to leverage the Swift open source project and collaborate in the OpenStack ecosystem for the engine of the SwiftStack product. Several early customers had considered building their own services from open source, and learned they could use SwiftStack for less than the cost of a fulltime employee, thus resulting in much lower cost than staffing a development team to build and operate open source software. OpenStack Swift is an engine that runs the world’s largest storage clouds, with many ultra-large-scale deployments in production. Currently, SwiftStack is the largest contributor to the OpenStack Swift project, on average contributing roughly 35% of code with each release with the majority contributed by an active developer community. With fewer engineers assigned to the core platform, SwiftStack has been able to focus up the stack on data management and operations for customers.

In operation, SwiftStack manages data across on-premises and public clouds through its unique out-of-band management service. That is, data is automatically placed based on policies selected by applications and users, not infrastructure. The service runs in the public cloud or on-premises, and supports multiple clouds. Moreover, it delivers provisioning, automation, visibility, cost optimization, vendor preference, access control, to name a few. Continued Blandini, “We offer Hybrid cloud storage, comprised of nodes in any private data center plus capacity from public cloud providers of our clients’ choice, scaling-out to hundreds of petabytes in a single namespace. At the highest level, SwiftStack software deployed on standard server hardware on-premises creates a storage cloud with similar characteristics to public cloud. Going further, SwiftStack provides a single namespace for all data, eliminating silos and offering the freedom to set policies to store data using capacity in public cloud buckets the same way it is controlled stored on-premises.”

For the M&E market, SwiftStack has more specific value in addressing business challenges. “We knew that our cloud architecture had the promise of helping with scalability needs in this industry, and over the last three years, we have worked with our clients to understand their existing workflow challenges and continually enhanced SwiftStack as a company and a software product to more specifically and completely address those needs.”

The development and pre-production phases are all about ideas and planning; SwiftStack isn’t very often involved at those stages—unless it’s in the context of planning and architecting for production. As a result, SwiftStack is usually deployed in the back end of the production phase through post-production and editing to the final steps of distribution and archiving.

Here are the six workflow areas where SwiftStack works best.

Media Asset Management
SwiftStack is natively integrated into leading MAM systems—providing petabyte-scale storage to keep client assets active and simplifying their environment by reducing the number of tiers in which their data lives.

Content ingested into SwiftStack is instantly available to any user—even to remote users thousands of miles away. Keeping all assets active in SwiftStack means users no longer have to wait for retrievals from LTO tapes or replication jobs to complete from other storage systems.

Multi-site Production and Active Archive
SwiftStack can be deployed in a single site or used to connect multiple sites together with a single view of the assets stored. Instead of using FTP or a WAN accelerator to send large files from point A to point B, with SwiftStack, any data uploaded at any site can be instantly accessed at any remote site. With SwiftStack, users can write anywhere and read everywhere.

SwiftStack storage is comprised of standard server hardware and is a cost-effective alternative to an LTO tape system. With SwiftStack, organizations can keep all their assets active—even in their archive—and eliminate the need for risky and time-consuming tape migrations.

If users require that their assets to live on multiple media types in multiple locations, they can connect their on-premises archive together with public cloud storage. SwiftStack’s Cloud Sync can automatically replicate some or all of their assets to Amazon or Google Cloud.

Ingest and Check-in/Check-out
The SwiftStack architecture makes it simple to scale network bandwidth in and out of the cluster. This allows it to provide massive throughput for uploading and downloading content, which means that it takes less time to store and retrieve big files. This is accomplished by moving lots of data in parallel—even saturating multiple 10Gb network links.

There are many ways to get data in and out of SwiftStack. Applications like asset managers and file transfer tools often use HTTP object storage APIs like S3 and Swift. End users can use the SwiftStack Client to browse assets in SwiftStack and upload or download them directly. SwiftStack also has SMB and NFS file protocols built into the core of the system to support traditional applications and workflows.

Distributed Transcoding and Rendering
SwiftStack allows for “ranged reads,” which means that several parts of a media file can be accessed in parallel. So, a transcoder that can run in containers like Docker or Kubernetes, results in many instances of that transcoder can also work on the same asset at the same time, which can result in dramatic time savings for large transcoding or rendering jobs.

Also, since SwiftStack storage can be extended into the public cloud using Cloud Sync, extra compute resources in AWS or Google Cloud can be utilized during a transcoding or rendering job. Defined by a policy, data can be automatically replicated to the public cloud, and results can be synced back on-premises when the jobs have completed.

Editing and Post-production
While most editing workflows utilize ultra-fast storage close to the editor (like the flash drive in your workstation), editing workflows are also beginning to leverage cloud storage technologies as well.

As an example, Vizrt’s Viz One MAM is tightly integrated with SwiftStack, and one feature they implemented is scrub editing of proxies directly in SwiftStack; with Viz One, editors don’t have to check out and check in media to work on proxies any longer.

Another example is Wiredrive, which essentially offers MAM-as-a-service. As editing occurs on dailies or other media, the asset is stored in SwiftStack and is accessible on mobile devices by directors or other remote staff who can comment or mark up the draft and provide near-real-time feedback to the craft editors.

Playout, Publish, and Transfer
If a studio’s final assets are stored in SwiftStack, then the organization’s active archive can also be the playout staging location for a solution like Snell. All assets are available to be moved to high-speed playout storage when needed.

For online publishing, platforms like Vimeo and YouTube often assume that the assets are uploaded from a local computer. With SwiftStack Drive, all assets stored in SwiftStack can be accessed like they’re on a locally connected drive.

If a studio’s workflow concludes with the transfer of assets to someone else, then using a tool like Aspera or Signiant might be the last step. They support sending and receiving data directly to and from S3-compatible storage, so studios can use them to transfer data from SwiftStack to wherever it needs to go. Or, users can provide external access to SwiftStack by sending a “temporary URL” to access a single file for a limited period of time.

The SwiftStack Technology Advantage
SwiftStack’s architecture is most similar to public cloud storage, so if you’re looking to build a storage cloud, but on private infrastructure behind your firewall with the capabilities to replicate and manage data on public cloud storage services, then SwiftStack’s Hybrid Cloud Storage architecture should be strongly considered. It delivers on the promise of software-defined storage by giving users the freedom to choose the hardware that best meets their needs with the ability to scale with something different. The SwiftStack solution allows users to pay-as-they-grow instead of being locked into yearly commitments with no visibility into what will happen next. And as expected with cloud storage, it’s ready for user applications today and will support those applications as they modernize over time, including when integrating with public clouds.

Earlier this month, the company unveiled SwiftStack 6, a multi-cloud data management offering that delivers a cloud-native, single namespace for unstructured data. Legacy applications with large unstructured data sets can now access and consume the same data from a single namespace via file protocols without the need for a gateway or application refactoring. This file access combined with SwiftStack Cloud Sync enables policy-based data placement, giving applications the freedom to run wherever IT deems the workload can run best, and without altering the user experience, all the while maintaining governance and control. SwiftStack 6 allows organizations to move beyond on-premises object storage to genuine multi-cloud data management that supports the cloud-first business initiatives that datacenters have been tasked with for the foreseeable future.

SwiftStack 6 gives users integrated File Access. Applications and users can now use SMB and/or NFS file access protocols to read and write to the single namespace, without a gateway. An important distinction of SwiftStack File Access is that data can be read and written in both formats. For example, ingest data via file, read via object, and distribute via file, and vice versa. Universal Access including Cloud Sync extends this capability of accessing unstructured data from any cloud location independent of the access method required by the user or application.

“In an IT future that’s becoming increasingly hybrid, the next phase of storage depends on the ability to manage data that will need to exist in multiple locations; both on and off-premises. Multi-cloud capable data management based on cloud-native technology can provide the framework, tools and resources necessary to deliver highly-available business data on a worldwide level,” said Steven Hill, Senior Analyst of Storage Technologies at 451 Research. “Traditional SAN and NAS systems will continue to maintain a role for the foreseeable future, but evolved and intelligent secondary storage—that’s capable of a deeper understanding of the value and relative importance of the information it protects—will become crucial to provide the policy-based data protection, automation, security, availability, cost-efficiency and granular lifecycle management needed to meet the challenges of next-generation business applications.”

Comprehensive Sales and Pricing Model
SwiftStack’s continued market expansion is due in part to the success of its sales and pricing model. The company identifies repeatable use cases and packages offers for installation with OEM and reseller partners, specifically Cisco Systems, which resells SwiftStack worldwide as a bundled solution with UCS servers via its sales organization and channel partners. These offers are more than reference architectures and packaged bundle SKUs, as they include sales enablement and execution-ready demand generation campaigns that help partners, such as Cisco, identify and engage prospects from their own audience. Some resellers, like Vizrt, do market their integrated solution using SwiftStack under the brand name of their specific storage server configurations.

SwiftStack is licensed on an annual term basis based on licensed capacity under management. SwiftStack’s licensing is the same whether using erasure codes or replicas for durability, and customers choose their durability and thus the amount of hardware required to deliver the usable capacity. This enables customers to purchase the license capacity they require at the time, and then add or reduce their licensed capacity on a pay-as-you-go basis. SwiftStack’s license also includes 24×7 production support and maintenance.

SwiftStack offers two licensing editions of the SwiftStack Hybrid Cloud Storage System – Enterprise and Enterprise Prime. Enterprise is for single-data center deployments while Enterprise Prime is inclusive of support for multi-datacenter replication and Cloud Sync between private and public clouds, such as Amazon Web Services (AWS) and Google Cloud Platform (GCP).

With the Enterprise Prime edition, customers pay a premium fee for multi data-center replication as well as Cloud Sync to AWS and/or Google Cloud Platform, which enables to SwiftStack to monetize data under management in both AWS and GCP.

Customer Validation with DreamWorks Animation
Of the many M&E customers that SwiftStack has in their client portfolio, DreamWorks Animation stands out as one of the best examples of how organizations are integrating their technology. Scott Miller, Senior Technologist for DreamWorks Animation describes the workflow process and what’s involved in creating a feature film.

Shrek, Puss in Boots, Kung Fu Panda, and the Madagascar animals are just some of the characters that have come to life thanks to the advanced CG technology at DreamWorks Animation.

However, animation is only one of many tasks involved in making a feature film. From story creation to the addition of 3D effects, animation feature films about 130,000 frames, made up of 250 billion pixels, requiring 75 million CPU hours performing more than two billion transactions, generating a half a billion files consuming 350 to 500 terabytes.

Depending on the complexity of a scene or sequence, each frame can include thousands of assets. An asset can be a file, a description of a file, or a collection of descriptions of files that define the relationships between all the elements of that frame, from fur to facial expressions to light and shadow and dozens more. Each asset has potentially a million files, and an asset can be requested or sent multiple times a day.

This massive workload relies on a shared distributed file system with a global namespace that can scale the capacity and delivery of data independently.

DreamWorks Animation has built its production asset management environment on software by SwiftStack – where the company’s solution provides the benefits and utility of public cloud storage within DreamWorks Animation’s own data centers in Southern California, Bangalore, and Shanghai. Along with elasticity to accommodate these assets, and these worldwide teams, DreamWorks gains data protection and active archive capabilities.

“Animation is a very collaborative sport,” noted Miller. “You need people working on the same data set at the same time, delivering content into the same repository for reuse by everybody else. As a nice handy side effect of working collaboratively, I get geographic durability, I get replication not just in different parts of the data center, I get them in different parts of the world.”

For DreamWorks Animation, the “product” is data, and it’s the result of both creative decisions and technology decisions.

“One of the things we’re trying to do as technologists is not be a barrier,” continued Miller. “We can give them some guard rails, and we can give them some guidance, but [we] don’t let creative ambition be stifled by the technology’s ability to deliver it.”

Future Opportunities
Looking ahead into the near future, Blandini noted that while SwiftStack already works with Vizrt, Primestream, and other MAM providers who support cloud APIs for storage, the ecosystem for MAM is not yet fully supporting cloud; hence there is great opportunity to broaden support by collaborating with MAM vendors to give customers increased choice. He also sees geospatial intelligence and the broader surveillance market are opportunities that SwiftStack is being brought into as it has similar very-large data sets.

For a more comprehensive look into SwiftStack and how it can help your organization save capex and opex costs in the new year, go visit





Contributing Editor at Broadcast Beat/AV Beat
Curtis Chan is President and CEO/Founder and Managing Partner of two high technology Brand Marketing and Public Relations companies, CHAN & ASSOCIATES and COGNITIVE IMPACT, whose clients range from the Global Fortune 500 to venture capital funded startups.

His agencies’ services, for over a quarter century, have helped garner early stage and growth companies significant brand exposure, resulting in either acquisition or IPO, with a total valuation of over $2.5 Billion.

Mr. Chan is a serial entrepreneur, philanthropist, author, mentor and angel investor. He played key executive roles early on in helping to pioneer and usher in both professional digital audio and digital video technologies for the recording, broadcast, cinematic and post production industries. He also played key executive roles in helping to develop and market data storage technologies. His expansive background spans four decades in ever increasing executive roles in operations, business development, engineering and sales/marketing in the information technology, media & entertainment, storage and networking, and other related high technology industries.

He is an active mentor and senior advisor to many established and VC backed start-up companies and incubators, a market/technology analyst, and has over three decades of experience in brand development, management consulting, company turnarounds, creative advertising and public relations.

A graduate of California Polytechnic State University (Cal Poly), San Luis Obispo; Mr. Chan recently ended his 14th year as Vice President for the Fullerton College Foundation; 22nd year as an Entrepreneur-in-Residence and mentor for the Small Business Institute at California State University Fullerton and incubators. Additionally, he is a volunteer member of DECA (Delta Epsilon Chi and Distributive Education Clubs of America); an advisor to the College of Engineering's Global Waste Research Institute, Cal Poly, San Luis Obispo and a member of the President’s Green and Gold Society; and advisory council member to Pepperdine University's Most Fundable Companies program. Passionate about hospice care, he was the past Board President for the Healing Hearts Association; and is a regularly featured speaker at many MBA and Doctorate level lectures around the country. He has presented and published over 30 papers worldwide, is a book co-author, an Editorial Board Member for the Society of Motion Picture and Television Engineers, past Senior Editor for Broadcast Beat and AV Beat Magazines, Computer Technology Review, and is a regularly featured Contributing Editor to many US and international trade publications for over 35 years.

View Cognitive Impact services at Curtis can be reached at; Office: (714) 620-4993
Curtis Chan
Broadcast Beat - Production Industry Resource