Dailies. B-Rolls. Circle-Takes. These digital video oriented processes of the “Hollywood” production marketplace have –for nearly a decade now– seen an explosion in the volume of recorded video that must be stored and managed. Entirely new workflows have been created to handle the deluge of video that digital movie-set cameras have unleashed. In the old days parts of movies, TV shows, and commercials would end up on the “cutting room floor” as sections of film were edited out of the production. Nowadays, every “take” is kept and possibly re-used in the bloopers edition or the director’s cut release.
SPOILER ALERT: This article is about the technical management and storage of surveillance video and not an article on the cool special-effects Hollywood is pretending to do with video surveillance.
What our colleagues in the very similar Media and Entertainment (M&E) –or Hollywood– marketplace have learned is how to manage this vast amount of (and significantly growing) recorded video that is generated every day. How do directors and producers quickly access and review today’s shoot? How do they select the takes and scenes that make it into a movie? How can they quickly and easily find video scenes previously recorded for other productions and reuse them in a new production? What’s the most cost effective, affordable, way to store all of these video assets? These are all questions the Hollywood marketplace has already figured out how to answer.
Therefore, with the IP video surveillance marketplace managing such a similar process to Hollywood in terms of storing vast amounts of recorded video, how come we have not embraced similar workflows in surveillance video lifecycle storage? Hollywood has already learned how to monetize recorded video over and over again.
Hollywood has already learned how to use tried-and-true I.T. storage technologies to store terabytes and petabytes of video at the lowest possible costs (especially operational costs). Hollywood has already learned how to add more information (metadata) to the recorded video to help make it more relevant for quicker searches in the future. Hollywood has already learned that if you can’t get back to the recorded video quickly and easily, it quickly becomes useless to retain it.
Many people in the M&E market will point to the introduction of the RED digital camera as the tipping point for Hollywood to move to a digital workflow. Originally conceived in 2005, RED became a driving force in moving away from film and now so much is recorded on digital medium. Removing the high cost of movie and television grade film allowed for production companies to “save it all” and not leave any video footage on the proverbial cutting room floor. One small side effect: there is A LOT of extra video.
Let’s examine two areas that Hollywood has gotten it right with regards to video management and video storage. The first is metadata. Metadata (data about the data) allows each user to add information about the video that can be later used to easily search and review the video. “Airplane flyover of Golden Gate Bridge at sunset” is a great example of a clip of video that a production company could easily reuse for that movie sequel about San Francisco. Airplane, golden gate bridge, and sunset are all terms (metadata) that would significantly help an editor quickly find this scene and use it again. Think of the cost savings this has over sending out another film crew to shoot another airplane flying over the famed bridge. Similarly, being able to pinpoint the specific clip of video in a video surveillance environment becomes easy using metadata. Metadata literally becomes worth its weight in gold.
Notice the similarities of the Hollywood market and the video surveillance market.
The other area that Hollywood has learned how to do it right is in the use of multiple tiers of storage for the recorded video. Just like in the video surveillance marketplace, video is not frequently reused or viewed after initial recording. And just like the video surveillance market, this rapid expansion of video assets is a relatively recent change and these newer solutions are now maturing. Therefore, what existing digital storage technologies are trustworthy enough to store our video assets and ensure they will be there when we need them? Enter LTO digital computer data tape.
LTO storage has been available since 2000 and has become the de facto standard in computer data tape storage and is heavily used in the Hollywood marketplace. The challenge for many in the video surveillance / security marketplace is that when they see the word “tape” they hear “VHS.” Although tape is indeed a four-letter word, if you continue reading this article with such a feeling you will miss out on all of its benefits.
IBM, HP and Seagate developed LTO to counter other data tape technologies thus introducing a more open format. Much of the technology is an extension of the work done by IBM at its Tucson lab during the previous 20 years with over 80% of the world’s data residing on data tape. Around the time of the release of LTO-1, Seagate’s magnetic tape division was spun off and eventually acquired by Quantum. Today, IBM, HP, Quantum, Spectra Logic, Oracle, along with a number of others manufacture LTO data tape libraries with IBM and HP manufacturing the LTO drives. The latest generation, LTO-6, began shipping in December of 2012 in 2.5-Terabyte cartridges with a current street price of about $90 per cartridge.
Adoption of LTO is already firmly established in the Hollywood media production environment. One of the driving forces behind this adoption is a mandate to many feature motion picture productions by insurance companies that content (video) captured on set or on location be archived to LTO tape on a daily basis. LTO meets the dual needs of the studios and the insurance bonding companies. The bonding companies feel safer and more at ease with digital because the content is archived on LTO, the same tape-based platform that banks use.
LTO is rated at up to 30 years archival shelf life. It provides for 5,000 cartridge loads/unloads and it allows for approximately 260 full file passes (with one full pass equal to writing enough data to fill an entire tape cartridge). And with the sequential data structure format of video, LTO data tape becomes an ideal storage medium.
An appropriate surveillance video workflow however must be put into place to properly utilize LTO as a video surveillance storage medium. Just like in the Hollywood market, there needs to be a way to reference the original video stored on LTO data tape without any prolonged delays. Just like Hollywood’s video editors, video surveillance operators cannot afford to wait to search and playback recorded video. By implementing the use of low-resolution companion files, these reference files can be stored on spinning hard disk and consume a far smaller file size (and thus storing a significantly longer period of time) and allow for quick and easy video playback.
Implementing sound metadata to our recorded surveillance video allows for more relevant search results and provides us a way to find the proverbial needle in our ever growing video storage haystack.
As in almost every video surveillance investigation, there is a fair level of forward or backward “scrubbing” to actually find the video of interest we are looking to review. These low-resolution companion files generate very low network bandwidth utilization with a great side effect of loading and playing very quickly. Low-resolution companion files allow us to bounce around our recorded video looking for what we seek with little overhead.
Then, just like a Hollywood editor, a surveillance video operator can select the snippet of video needed to deliver to HR/Police/Court/etc with the companion file timestamp directing the system to the specific original, untouched – unaltered, video clip located on the specific LTO cartridge. This taking but a few minutes of transfer time to complete. This best-practices approach to multi-tiered video storage also delivers appropriate “Chain-of-Custody” needed to submit the video as evidence in a court of law. However, it’s the speed that surveillance video operators will care about most with the ability to quickly find the video they are interested in.
From a cost standpoint, this becomes a more superior infrastructure. Compare the acquisition costs of terabytes and petabytes of spinning NAS and SAN and then the cost of an appropriate LTO data tape library, complete with robotics. The LTO implementation is significantly less expensive. Adding in the 3- to 4-year lifespan of spinning disks compared to the longevity of LTO incurs another maintenance cost. Then throw in the ongoing monthly operational costs of electricity for all that spinning disk and the cooling costs to keep those SAN/NAS units spinning. 24/7. Year round. Those costs add up quickly.
Implementing a sound multi-tiered video storage infrastructure with spinning disk for our low-resolution companion files linked to our LTO stored –unaltered– original video allows for more video to be stored for less. How do we create a disk-and-LTO infrastructure to provide us what we need in video storage, without breaking the bank?
When contemplating a 21st century surveillance video management solution, we realize that a spinning disk-only approach harkens us back to pre-Columbus days. There are better ways to reach our destination. We all want to store as much video as we can. We want to record at the highest resolutions. We want to record with the highest frame rates. And we want it as affordable as possible. So why not learn from our colleague in the Hollywood market?
I am confident that the video surveillance market will indeed adopt these workflow concepts from our colleagues in Hollywood and you will see increasing usage of metadata, low-resolution companion files, and LTO storage devices. Embracing the sequential data structure format of video along with the utilization of LTO data tape delivers an ideal storage solution. And done properly the implementation of LTO multi-tiered storage surveillance video recording is actually the perfect “killer app.”
Thanks Hollywood.
Resources:
Wikipedia defines Linear Tape-Open:
http://en.wikipedia.org/wiki/Linear_Tape-Open
SpectraLogic LTO robotic libraries
http://www.spectralogic.com/index.cfm?fuseaction=products.displayProductSummaries&catID=145&src=fly
Low cost digital storage fuels new video business strategies:
Low-Cost Digital Storage Fuels New Video Business Strategies
Codex integrates LTO in media management solution:
http://news.creativecow.net/story/871666
Fox implements Dalet LTO content management:
http://www.dalet.com/press-release/fox-implements-dalet-enterprise-edition-sports-content-management