Tech Corner – Harry B. Miller III, ACE
CE 2020 1st Qtr
Start your day with a heart attack by opening your master edit sequence and see the following….
Yes, all that red color means that everything in this timeline is offline. EVERYTHING.
It then occurs to you that in the process of wrapping a documentary film (which is mirrored on two hard drives), that you deleted a terabyte of unneeded media. Well, you thought it was unneeded. Ack! That big chunk you threw into the trash wasn’t unneeded.
It was, in fact, the online media that had been consolidated and copied to your Master drive for when there will be a re-edit for fixes and a shorter version of this movie. And in order to use that space, you emptied the trash. Ooops. That was stupid. Now what? This documentary had been onlined and delivered for screening. Over the course of nearly three years of work, two hard drives had expanded to three, and there were gigabytes of Avid media files no longer needed. A “housecleaning” seemed in order. At the time that seemed like a good idea. The 12TB Master drive held the camera original media, still photos, archival footage and the original transcoded Avid media files.
The 4TB portable Work drive held Avid media file duplicates and the project. Eventually a third drive was needed to hold the overflow of Avid files, backups and old outputs. In clearing out space on the smaller Work drive, I used a method from Kevin P. McAuliffe’s YouTube tutorial series ‘Let’s Edit with Media Composer.’ In the lesson titled “Archiving Your Project,” Kevin shows how you can reveal in Avid’s Media Tool exactly what clips are used in your timeline.
If you reverse that selection, you have then highlighted every clip of media that is not in your timeline. At that point you could hit DELETE and all those clips would be gone from your hard drive. But that may have unintended consequences. Rather than deleting everything not in the timeline of the locked cut, I chose to hide files that I was pretty sure were no longer used, mostly files from early on in the project when everything was digitized.
I used the Media Tools list of files not in the cut to find folders of media that were large and seemed to have the most unnecessary files. Those folders were moved into hiding. Anything that went offline I could find in the hidden files and move back into the project. Everything else could be deleted.
This went extremely well, where I recovered nearly a terabyte of space on a 4TB drive. The Master drive was also nearly full, but the Avid media files folders contained just that, work files, not high-resolution masters. So I overwrote several folders of media on the Master drive with folders from the Work drive.
As the new folders were smaller, it recovered lots of space on the 12TB drive. What I forgot is that in the process of onlining the two-hour documentary, the post house had built a NEW timeline with the final online media, graded and mostly DNxHD 175. I had overwritten all of that high-resolution media. The online timeline was now waaaay offline. Knowing a little about hard drives, I understood that even though a file has been deleted and the Trash cleared, the drive sectors that held that information retain it until those sectors are overwritten with new data. So, there was at least a chance that some of those files could be recovered.
But how? What tool can do that? Off to the internet! Searching for a specific answer to a technical question on the web requires wading through a lot of junk. The first few results are always ads to buy something, but then you can find technical articles that begin to answer the question. The first few suggestions are generally absurdly obvious (“First, don’t empty the trash”). And then I found the answer: Disk Drill.
Disk Drill 3 by CleverFiles is a drive utility, data recovery software for Mac and Windows. It is a free download (aren’t they all), with which you can scan the drive for missing files. If you want to get those files back you pay a $90 license.
It takes over an hour for Disk Drill to scan a 12TB hard drive. But amazingly it looked like the chances of recovering useful media were quite possible.
Because of the way Avid Media Composer works, newly imported media always goes into a MXF folder named “1.” I could be reasonably sure that all the missing files I needed would be in the “1” folder on the Master drive.
Disk Drill indicated after the scan that it found close to 500GB of recoverable files that were once in the “1” folder. My chances for recovery of my master media were looking better. (Pictured here is the list after files were recovered.) So I’m in for the $90 upgrade.
An important consideration when recovering files is to have a separate drive to put those files. Any writing to the Master drive could destroy recoverable files. After another two hours of copying the hidden-but-recoverable files to another drive, I was able to open the Media Composer timeline with the following result.
Not perfect, missing maybe 30 files from a timeline of hundreds of clips. All in all a great success. And the next time I want to “do a littlehousecleaning,” I think I’ll just buy another hard drive.
It would be simpler and likely cheaper than going “oops” again.