A newly launched company, TransferSoft (www.transfersoft.com), is introducing a revolutionary way to move data between systems and around the globe. This is noteworthy for a multitude of industries because the global 
enterprise market has seen explosive data growth over the past years as a result of the evolving landscape in the M&E, big data, social media, cloud and mobile markets, to name a few. IDC estimates yearly enterprise storage compound annual growth rates (CAGR) at over 50%, with traditional applications in IT departments growing data at over 30% yearly. This exponential data growth has resulted in a corresponding increase in storage subsystem capacity and the need for ever increasing LAN, WAN and GAN network bandwidth to move, replicate, backup, and copy this data onsite, across the country or around the world.
What is HyperTransfer?
For the broadcast, digital cinema and video centric industries – hardware, software and hybrid systems currently used to move files onsite, across the country, or around the world have had incremental progress, until now. TransferSoft has developed HyperTransferTM, a unique and embeddable software solution to address this challenge. Aside from providing higher sustained throughput, HyperTransfer also takes into account how existing applications are structured, the desire to operate on data inflight without affecting performance, how bandwidth and processing will advance, and the realities of securing, operating on, and protecting data as it moves from point to point or around the globe.
Allan Ignatin, president, CEO, and co-founder of TransferSoft commented about their newly developed technology stating, “HyperTransfer is a ground-breaking technology suite of embeddable software tools for OEMs designed to fill the holes created by aging transfer mechanisms that have failed to keep up with the relentless growth and need to move structured and unstructured data across networks.” According to Ignatin, TransferSoft’s HyperTransfer technology is designed to be easily embedded into a wide range of appliances today without the need to write a single line of code; capable of up to 40Gb/s or 5 GB/s transfer speeds, and can easily scale in parallel with future network and processing advancements. HyperTransfer is also industry agnostic in that it allows users to move their data at full throttle speeds no matter the application or industry – whether it’s to replicate files across the LAN, WAN or GAN, migrate data, converge data centers, or have industry specific needs, such as in Media and Entertainment, Life Sciences, Energy, Government/DoD, or Financial Services.
Currently, studios and vendors needing to move large content files from point-to-point or point-to-multipoint are forced to manually integrate costly data transfer software and appliances that buckle under the weight of data transformation operations. “By reimagining data movement, our patent pending technology not only increases the speed and efficiency of data transfer but also operates on the data during transmission,” continued Ignatin. HyperTransfer implements data transforms (data compression, de-duplication, encryption, and statistical differencing), in virtually any combination, during data movement. All of these features operate at full bandwidth, independent of latency introduced by the distance between the source and destination services.
Looking Under the Hood for the Secret Sauce
We can break TransferSoft’s claims into three distinct areas: 1) Data Movement Links; 1) Transforms; and 3) HyperTransfer Technology.
Links: Abstracting LAN or WAN Connections
Links are at the heart of implementing HyperTransfer as a service in an application and defines how files are treated between the source and destination. Each Link is configured with a set of capabilities including describing the source and destination service, the additional data transforms to be employed, and can include the file system directories or lists of files to be transferred.
Links are Persistent
Each Link can be configured with a set of capabilities including describing the source and destination service, the additional data transforms to be employed, and may optionally include the file system directories or lists of files to be transferred. Links also may define how files are treated between the source and destination. Settings allow Overwrite Always, Overwrite if Newer, and Overwrite Never. Links are persistent, but use no CPU resources while not in use.
Virtual Links
When services have no direct LAN/WAN connection to a desired destination service, HyperTransfer provides the ability to create Virtual Links by routing data through existing Links without sacrificing performance. For example, if Site A is connected to Sites B and D; and Site B is also connected to Site E, even though Sites D and E have no direct connection, HyperTransfer can establish a Virtual Link between D and E automatically routing the data between A and B. Performance is delivered as if D and E had a direct Link only limited by the speed of the slowest link in the chain. See the diagram below.
Built-in GUI and Restful API
Links may be created though the supplied GUI or OEM derivations of the GUI, or by RESTful (Representational State Transfer (REST) is a software architecture) use of the API. Links also support the use of source and destination pipes for OEM-defined processing of both file- and block-based data.
Link features include:
- Bi-directional providing simultaneous data movement in each direction
- Both directions run at full available bandwidth up to 10GbE
- Persistent using no CPU resources when not in use
- No limit to the number of Links that may be established and maintained
- Moves millions of tiny files at the same speed as one or more terabyte-sized files
- Support the use of source and destination pipes
- Processes both file and block data
- Files can move while they are still being created allowing them to be usable at the destination within seconds of closing the write function at the source
- Multiple Links can execute simultaneously
- Virtual Links route data to a destination with no direct LAN/WAN connection through existing Links without sacrificing performance
- Settings allow Overwrite Always, Overwrite if Newer, and Overwrite Never
- Reorganizes data packets into their original sequence before writing at the destination to reduce disk latency and the read
Transforms: Acting on Data Inflight without Loss of Transmission Speed
HyperTransfer has the unique ability to act on or transform data as it is being transferred while continuing to 
operate at full bandwidth, independent of latency introduced by the distance between the source and destination services. This capability is exclusive to TransferSoft providing the ability for OEMs to accelerate applications and solutions they deliver to their end customers. HyperTransfer implements the following data transforms, in virtually any combination, during data movement:
Data Transforms
HyperTransfer implements the following data transforms, in virtually any combination, during data movement. All of these features operate at full bandwidth, independent of latency introduced by the distance between the source and destination services
Block-by-block Checksums
Checksums provide verification that the data transmitted is the data received. Supported checksum algorithms include MD5, SHA2, and SHA3.
Data Compression
Two data compression algorithms are provided: Compression A for higher compression at slightly higher CPU utilization, designed primarily for Links with bandwidth under 300Mb/s; and Compression B, designed primarily for Links with bandwidth between 300Mb/s up to 10GbE, with extremely high performance and slightly lower compression efficiency. All compression operations are auto-sensing; the algorithms determine whether the current data passing through the transport is compressible, and if not, turns compression off to save CPU cycles; when compressible data is encountered, compression operations are automatically re-enabled. Acceptable compression levels (for example 1.5:1) may be set, in which case compression will not be enabled unless the data passing through the transport is compressible to the selected level.
Encryption
Two encryption algorithms and five alternative selections are provided. Available are AES CTR-mode or AES GCM-mode, both supporting either 128- or 256-bit key lengths. Also available is the TransferSoft proprietary RVX algorithm, with higher security by means of greater entropy, and with the MXC function of the RVX algorithm which destroys the keys instantly upon use. Enough RVX keys are created in the first few milliseconds of RVX operation to encrypt a data set of approximately 15 petabytes.
Statistical Delta Differencing
These statistically-based de-duplication functions may be employed in four scenarios:
- Static Mode – a statistical “map” of a file(s) is maintained at the source, compared with a map created at the destination while files are being transferred, and only data that has changed is transferred. The destination files are updated in place, and the source Static “map” is updated to reflect the current state of the file(s).
- On-the-Fly Mode – as files are transferred, differences between the source and destination data is identified and only the changed data is transferred. HyperTransfer is orders of magnitude faster at high latency than the popular utility
- Cumulative Mode – as directories of files are transferred, a cumulative map is maintained so that any data chunk that has previously been encountered will result in the transfer of only the reference to the data at the destination. Thus, data transfer volume is reduced by taking advantage of previously encountered data, regardless of the file in which it originally resided.
- Stream Mode (Statistical Compression) – a unique method to de-duplicate data that has not previously been transferred. In a manner similar to Cumulative Mode (above), a statistically-derived map of data sent during the process is used to identify data chunks that have previously passed through the stream, and only a reference to the destination location and length of the data is transferred, saving bandwidth by significantly reducing data volume on the network. For example, the TransferSoft team demonstrated that they can de-dup a file on itself by 60% on the first transmission while compressing the rest – all in real time.
HyperTransfer Technology: Easy Implementation
The HyperTransfer technology suite is enabled via a REST interface for easy integration into any product built on major operating system platforms including Linux/UNIX/AIX, Windows, and MAC OS X. An embedded web server provides the REST interface for bringing XML statements to the surface of any new or existing GUI. A fully implemented GUI, including source code, is also provided with licensed HyperTransfer software.
With a very small footprint, the PowerPort™ technology allows nearly instantaneous embedding of HyperTransfer to any TCP program without the need to write a single line of code.
HyperTransfer runs as a service and is quick to install at source and destination locations. Once installed, the HyperTransfer GUI or API is used to create Links and then by pointing a program to the Links, the embedding is complete. No new code of any kind is needed. The finer points of the technology are as follows:
Runs as a Service
HyperTransfer runs as a service on two or more systems. There is no limit to the number of services that may be linked together for a fully integrated data movement environment.
Fully Documented and Open RESTful API
The HyperTransfer service features are enabled via a REST interface for easy integration into any product. An embedded web server provides the REST interface for bringing XML statements to the surface of any new or existing GUI.
Graphical User Interface
A fully implemented GUI, including source code, is provided to OEMs along with licensed HyperTransfer software. The GUI, with QT-enabled graphics, includes the underlying C++ code as a guide for implementation, or for cut and paste into OEM-created or existing GUIs. The GUI can be branded for OEMs with minor effort, or by TransferSoft™ for a modest NRE charge.
No Bandwidth Limitations
HyperTransfer is delivered with no bandwidth limitations and supports full bandwidth bidirectional data movement up to 10GbE at no additional cost.
Data Movement Independent of File or Block Size
Unlike other hardware or software products, HyperTransfer moves millions of tiny files at the same speed as one or more terabyte-sized files. On a 10GbE connection, a 650GB file transfers at 1GB/s. On the same 10GbE connection, a directory structure of 1,000,000 files averaging 64KB in size, in 16-deep nested folders, will transfer at 1GB/s.
InstaTransfer™
HyperTransfer allows the movement of files while they are still being created (sometimes referred to as “growing files”). This allows files to be usable at the destination within seconds of closing the write function at the source.
Strict Sequence of Transferred Data
In addition to employing TCP as the underlying transport technology, and even with multiple data nodes (often called “streams”) in single or simultaneous transfers, HyperTransfer reorganizes data packets into their original sequence before writing at the destination. This reduces disk latency and the read overhead associated with scattered subsequent read functions.
Data Movement Reimagined
TransferSoft’s HyperTransfer technology is a significant milestone, if proved successful, will empower the data storage and networking OEM community to quickly and cost effectively incorporate this embeddable technology into the next release of their solution and also readily make it available to their installed base of customers. For the broadcast, video and digital film/cinema centric industries where gigabyte to terabyte sized files are constantly being transferred across the country and across the world, HyperTransfer’s tremendous efficiency allows OEMs to use less costly hardware without any sacrifice in performance, reliability and data integrity. Moreover, the cost savings in the reduction of transfer time per file for a given bandwidth, and the resultant benefit of being able to send more files in the same timeframe is a win-win for all.
One of TransferSoft’s first partners is JMR Electronics (www.jmr.com), a global storage vendor and systems integrator. Their first appliance product, Net Accelerator, embedded with TransferSoft’s technology will be showcased at this year’s NAB show in booth SL12117.
His agencies’ services, for over a quarter century, have helped garner early stage and growth companies significant brand exposure, resulting in either acquisition or IPO, with a total valuation of over $2.5 Billion.
Mr. Chan is a serial entrepreneur, philanthropist, author, mentor and angel investor. He played key executive roles early on in helping to pioneer and usher in both professional digital audio and digital video technologies for the recording, broadcast, cinematic and post production industries. He also played key executive roles in helping to develop and market data storage technologies. His expansive background spans four decades in ever increasing executive roles in operations, business development, engineering and sales/marketing in the information technology, media & entertainment, storage and networking, and other related high technology industries.
He is an active mentor and senior advisor to many established and VC backed start-up companies and incubators, a market/technology analyst, and has over three decades of experience in brand development, management consulting, company turnarounds, creative advertising and public relations.
A graduate of California Polytechnic State University (Cal Poly), San Luis Obispo; Mr. Chan recently ended his 14th year as Vice President for the Fullerton College Foundation; 22nd year as an Entrepreneur-in-Residence and mentor for the Small Business Institute at California State University Fullerton and incubators. Additionally, he is a volunteer member of DECA (Delta Epsilon Chi and Distributive Education Clubs of America); an advisor to the College of Engineering's Global Waste Research Institute, Cal Poly, San Luis Obispo and a member of the President’s Green and Gold Society; and advisory council member to Pepperdine University's Most Fundable Companies program. Passionate about hospice care, he was the past Board President for the Healing Hearts Association; and is a regularly featured speaker at many MBA and Doctorate level lectures around the country. He has presented and published over 30 papers worldwide, is a book co-author, an Editorial Board Member for the Society of Motion Picture and Television Engineers, past Senior Editor for Broadcast Beat and AV Beat Magazines, Computer Technology Review, and is a regularly featured Contributing Editor to many US and international trade publications for over 35 years.
View Cognitive Impact services at www.cognitiveimpact.com. Curtis can be reached at [email protected]; Office: (714) 620-4993
- Vivaro Media Selected as the Official Broadcast Distribution Partnerfor the CIC Mont Ventoux Professional Cycling Race - May 30, 2023
- Axle AI launches AI-powered MAM for OpenDrives storage platform at NAB - April 18, 2023
- Matrox Blueprints Next-Generation On-Premises and Cloud-Based Live Production Workflows at IBC 2022 - September 9, 2022
