Guidance on estimating storage requirements for digital audio and video

[Creighton Barrett]

Hi everyone,

We recently completed a digital archives collection assessment and now have a register of digital media carriers that require prioritization, processing, etc. We're working on the report for this project and would like to provide some reasonably accurate estimates of digital storage requirements. Our approach has been to indicate, for example, that a CD-R can hold up to 700 MB or a standard 3.5 floppy can hold up to 1.44 MB. And then multiply by the number of media carriers to provide an estimated "maximum storage requirements" for that format. Not perfect, but it works, sort of.

The challenge comes with formats like DVCAM, MiniDV, DVCPRO HD, etc. The storage requirements for digital video vary widely depending on certain criteria that was beyond the scope of our assessment project (we have not yet imaged or examined the carriers). Storage requirements are also affected by yet-to-be determined policies and technical specs for transferring and storing HD video.

To give an example: the method of estimating storage requirements for a given format has, so far, told us that we have about 7 TB of digital material on these carriers. But we have 481 DVCAM tapes, for example, and no estimated maximum storage requirement. One test we did found that a DVCAM with one hour of HD video produced a 10-bit uncompressed MOV file at 707 GB. So we are potentially looking at hundreds of additional TB.

Any ideas? How have you approached the problem of estimating storage requirements? Do you have a magic number you use for certain formats? Or a sampling method that you have used to extrapolate storage requirements for a larger set of digital media carriers?

I would appreciate any thoughts, suggestions, references, etc.

Thanks,

Creighton

[Heidi Elaine Dowding]

This is slightly outdated, but this is a report that has led to Indiana University's massive Media Digitization and Preservation Initiative (leading to 6.5 PB of preserved obsolete media): http://www.indiana.edu/~medpres/documents/iub_media_preservation_survey_FINALwww.pdf

I wasn't involved in the initial media survey, but you would likely get really good information from Mike Casey (mic...@indiana.edu), he heads the MDPI project.

All Best,
Heidi Kelly

--
You received this message because you are subscribed to the Google Groups "Digital Curation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-curation+unsubscribe@googlegroups.com.
To post to this group, send email to digital-curation@googlegroups.com.
Visit this group at https://groups.google.com/group/digital-curation.
For more options, visit https://groups.google.com/d/optout.

--

Heidi Elaine Kelly (she/her/hers)

dowd...@gmail.com | heid...@indiana.edu

[Bertram Lyons]

Hi Creighton --

Forwarding this on behalf of my colleague at AVPreserve, Rebecca Chandler (reb...@avpreserve.com). She isn't currently signed up to post on this list:

<begin>

Hi Creighton,

DV tapes will have been recorded using the native DV codec. As the codec is already compressed, transcoding to uncompressed will add nothing to the quality of the image and digitizing as with other video formats will result in the loss of metadata contained in the bitstream. If I know nothing about how a DV tape was originally recorded, I assume the following:

miniDV, DVCAM, DVCPRO = 25 Mbps

DVCPRO HD = 100 Mbps

Keep in mind, these numbers reflect video ONLY. Audio could have been recorded in a few different ways; I usually use the largest option so as to be over prepared:

miniDV, DVCAM, DVCPRO = 1.5 Mbps for a stereo file

DVCPRO HD can support up to 8 channels, so if you want to be extra careful = 6 Mbps

There will be a few that don't fall in line with these assumptions, but it's usually fairly accurate. A good resource showing the data rates of different DV codecs can be found here: https://documentation.apple.com/en/motion/usermanual/index.html#chapter=B%26section=2%26tasks=true

</end>

Best regards --

Bert

______________________________________
 
Bertram Lyons, CA
AVPreserve
634 W. Main St., Ste 202
Madison, Wisconsin 53703
 
office: 202-430-4457

https://www.avpreserve.com 
Facebook.com/AVPreserve
twitter.com/AVPreserve

--

[Creighton Barrett]

Thank you Heidi and Bertam, for your responses. Sorry for the delay, but both responses were very helpful!

Bertram, we know that there is nothing to gain from transcoding compressed video to uncompressed for preservation purposes. But we're not comfortable assuming that everything on our DV tapes was recorded with a standard def DV codec. Most of these tapes come from a local professional film production company and an artist-run centre. These organizations were often (but not always) using HD cameras that can record HD video on DV tapes. And there are many MiniDV tapes, for example, that are clearly marked "HDV." These are Sony labels with check boxes for DV, DVCAM, and HDV.

In fact, the tape that produced a 707 GB uncompressed file was a Sony DVCAM MiniDV but the label indicates that the video was recorded in HDV. This video was transferred a while ago, before we had a good sense of this material, but we asked our vendor to produce two different preservation masters so we could start preservation planning for digital video. We got an HD 1080i QuickTime file and a 10-bit QuickTime file that was down converted and letterboxed.

So, I am now assuming that a video recorded in an (unknown?) uncompressed codec was transferred into a MOV container with an uncompressed codec (BlackMagic v210 YUV) and the result is a 707 GB HD MOV file with about an hour of video and audio. I'm just doing rough calculations right now, but it seems like the video storage requirements are more like 165.75 Mbps. The down converted file is 101.1 GB, or approximately 28 Mbps. But it was transferred into a MOV container with the same BlackMagic codec.

The 1080i file is much larger than any estimates I can find for digital video, including Rebecca's which are noticeably higher than the figures on the Apple site. I am a little fuzzy on the relationship between HD video and compression in video codecs - is HD video always uncompressed?

I guess we need to find out what the file size would be if we transferred the same video with a Sony HDCAM codec? Maybe the v210 codec produces significantly larger files?

Thanks again for your responses!

Cheers,

Creighton

[Simon Spero]

HDV is MPEG-2. 

"HDV video and audio are encoded in digital form, using lossy interframe compression. Video is encoded with the H.262/MPEG-2 Part 2 compression scheme, using 8-bit chroma and luma samples with 4:2:0 chroma subsampling. Stereo audio is encoded with the MPEG-1 Layer 2 compression scheme. The compressed audio and video are multiplexed into an MPEG-2 transport stream, which is typically recorded onto magnetic tape, but can also be stored in a computer file."

See https://en.wikipedia.org/wiki/HDV

Simon

[Michael Kjörling]

On 12 Dec 2016 12:28 -0400, from csba...@gmail.com (Creighton Barrett):

> So, I am now assuming that a video recorded in an (unknown?) uncompressed
> codec was transferred into a MOV container with an uncompressed codec
> (BlackMagic v210 YUV) and the result is a 707 GB HD MOV file with about an
> hour of video and audio. I'm just doing rough calculations right now, but
> it seems like the video storage requirements are more like 165.75 Mbps. The
> down converted file is 101.1 GB, or approximately 28 Mbps. But it was
> transferred into a MOV container with the same BlackMagic codec.

1080x1920 px is just over 2 Mpx. At 24 bpp, it's 5.93 MiB of image
pixels per frame. 25 frames per second gives about 148 MiB per second
(adjust as needed if the video was recorded at a different frame
rate). There is 3600 seconds in one hour, so with absolutely no video
compression whatsoever and 1080 16:9 video at 25 fps, this would
result in about 534 GiB of video data per hour.

High quality audio is at most a few gigabytes per hour (a good rule of
thumb is to assume 10 MB per minute for uncompressed 44.1 kHz 16-bit
stereo "CD quality"; I believe DVD uses 48 kHz 16-bit with a minimum
of two channels, which may provide a good baseline) so is basically
noise at these amounts.

An absolutely uncompressed 1080 16:9 video file should be closer to
550 GiB than 700 GiB. Half that if the video encoder can take
advantage of the fact that the video is interlaced, so for effectively
uncompressed 1080i, mathematically, I would _expect_ to see something
on the order of 300 GiB per hour. Something feels amiss if you are
seeing something closer to 700 GB per hour.

--
Michael Kjörling • https://michael.kjorling.se • mic...@kjorling.se
“People who think they know everything really annoy
those of us who know we don’t.” (Bjarne Stroustrup)

[Creighton Barrett]

Thanks once again, and sorry for the delay. Michael, I think you are right about something being amiss with this test. Just to clarify though, it looks to me like the frame size for HDV is 1440 x 1080 not 1920 x 1080, so the smaller number of pixels per frame would affect the overall calculation you describe, right? I'm not quite sure where you're rounding, but I think a frame at 1440 x 1080 would be more like 1.5 Mpx.

The MiniDV tape was recorded in HDV 1080i, so we should be expecting a very large uncompressed video file. But the 707 GB file is 1920 x 1080 not 1440 x 1080. And the v210 codec is 10-bit uncompressed but HDV is 8-bit compressed. Even if we wanted to store all of our digital video uncompressed (and we do not), I assume that both of those facts would contribute to a larger-than-necessary file size, right?

We only have one example like this. It was a test done with a vendor a few years ago to help us set some technical specs for preservation of digital video. So it's not a problem, but I don't think we can rely on the sample to extrapolate storage requirements for all of our digital video.

We're digging into this now and have policy questions about transferring digital video from a tape to a server that we could not ask when the test was done. Like:

• How do you choose between firewire or SDI?
• Why and when could you use a 10-bit codec to transfer digital video that was encoded in 8-bit?
• How should you choose the codec for each digital file?
• How should you choose the resolution for each digital file?
• How should you choose the wrapper for each digital file?
  

It seems that these policy questions must be answered before one can make an informed estimate of storage requirements. We want this estimate because it is a critical part of our determining the overall sustainability of digital video in our institution. And it sounds like, if you want a reasonably accurate estimate, you need to have detailed information about the recording mode and duration of footage on each tape as well as policies for how to handle the various scenarios (e.g., DV, DVCAM, 720 HDV, 1080i HDV, etc.). Sigh.

We are working with these tapes to start gathering more accurate information about recording mode, duration, etc. but in the meantime, I remain interested in other examples of how folks have estimated storage requirements for audio and video stored on media carriers. Thanks again for everyone's assistance!

Cheers,

Creighton

[Michael Kjörling]

On 15 Dec 2016 14:12 -0400, from csba...@gmail.com (Creighton Barrett):

> Thanks once again, and sorry for the delay. Michael, I think you are right
> about something being amiss with this test. Just to clarify though, it
> looks to me like the frame size for HDV is 1440 x 1080 not 1920 x 1080, so
> the smaller number of pixels per frame would affect the overall calculation
> you describe, right?

Sure, but it only makes the situation weirder, because with less
pixels, for uncompressed video, the file should be _smaller_. (Less
data that needs to be stored.) 1440x1080 (1.56 Mpx) instead of
1920x1080 (2.07 Mpx) should give 75% of the storage requirement, so
about 4.45 MiB per frame or about 391 GiB/hour (instead of 534 GiB/h)
at 24 bits per pixel (bpp).

> I'm not quite sure where you're rounding, but I think
> a frame at 1440 x 1080 would be more like 1.5 Mpx.

I wasn't rounding while calculating, but did round the figures that I
put in my e-mail for convenience. You can follow along in my original
e-mail multiplying to get the exact numbers if you like.

> The MiniDV tape was recorded in HDV 1080i, so we should be expecting a very
> large uncompressed video file. But the 707 GB file is 1920 x 1080 not 1440
> x 1080. And the v210 codec is 10-bit uncompressed but HDV is 8-bit
> compressed.

10-bit instead of 8-bit per channel would lead to an increase in file
size of 25%. (You are using 30 bits per pixel instead of 24 bits per
pixel, even if 6 of those bits per pixel are effectively wasted.
Dividing 30 by 24 yields a ratio of 1.25.)

Ignoring compression, 1440x1080 at 30 bpp needs about 5695 KiB per
frame (139 MiB per second at 25 fps, 489 GiB/hour), and 1920x1080 at
24 bpp needs 6075 KiB per frame (correspondingly 148 MiB per second,
521 GiB/hour), so those two almost cancel each other out, with
1440x1080x30bpp requiring slightly less storage than 1920x1080x24bpp.

1920x1080 at 30 bpp is 7594 KiB per frame or about 652 GiB/hour,
though, which brings us into the ballpark of your original "707 GB for
about an hour".

> Even if we wanted to store all of our digital video
> uncompressed (and we do not), I assume that both of those facts would
> contribute to a larger-than-necessary file size, right?

Oh, I'm not saying that you should be storing the video completely
uncompressed; there's _tons_ of data in any stretch of video that can
be compressed away (and especially at reasonable bit rates, modern
video codecs are really good at preserving the visual experience).

What I am saying is that the amount of uncompressed (or decompressed)
video data places an upper bound at what you should be seeing in terms
of file size. There is no reason whatsoever for a video codec to store
significantly _more_ data than what would be needed to represent each
frame as a separate uncompressed image with the same fidelity.

[Simon Spero]

On Dec 15, 2016 1:12 PM, "Creighton Barrett" <csba...@gmail.com> wrote:

The MiniDV tape was recorded in HDV 1080i, so we should be expecting a very large uncompressed video file. But the 707 GB file is 1920 x 1080 not 1440 x 1080. And the v210 codec is 10-bit uncompressed but HDV is 8-bit compressed.

1440 x 1080 frames are squished, and are expanded for display,so one would might expect this for the uncompressed video. 

• How do you choose between firewire or SDI?

Firewire. The SDI output is doing all the mangling on the camera, and may introduce artifacts (especially if there is error recovery). 

• Why and when could you use a 10-bit codec to transfer digital video that was encoded in 8-bit?

For better compression! With h264 10 bit color gives better results for the same bandwidth than 8 bit.

 http://x264.nl/x264/10bit_02-ateme-why_does_10bit_save_bandwidth.pdf

• How should you choose the codec for each digital file?
• How should you choose the resolution for each digital file?
• How should you choose the wrapper for each digital file?
  

For lossless archival copies, sticking with the original bits is probably the safest option. Wrapper format may not  matter too much. 

At the moment, for access copies, h264 in mpeg,  at the usual resolutions, and at bandwidths that allow for sufficient access, and/or  preserve sufficient quality for expected uses, such that that the expected cost of increasing quality exceeds expected cost of accessing the archival copy. h265 encoders don't seem to be there yet. 

Compression time is a consideration; however multiple jobs can be run in parallel, either on your own server,  or on a campus or departmental cluster. 

For estimating the space requirements for storing the contents of mini-dv tapes in native format, one can rely on the characteristics of the physical medium. An upper bound of about 13GB / hr is a reasonable estimate. 

[Creighton Barrett]

Thanks, Michael and Simon. Re: the difference between 1440 x 1080 recording and 1920 x 1080 display is sort of explained here: https://documentation.apple.com/en/finalcutpro/professionalformatsandworkflows/index.html#chapter=2%26section=6%26tasks=true

Simon, I think you're right - The native frame size for HDV is 1440 x 1080 and the frames are "pillarboxed" so they can be displayed at 1920 x 1080. I suppose this means that the extremely large MOV file is along the lines of what we should expect *if* we decided, for some reason, to preserve HDV 1080i video as 10-bit uncompressed video. Which we are not.

In any case, this has all been very interesting research for us! We are very close to having (a) some technical guidelines for DV video transfers and (b) some formulas for estimating storage requirements for the major digital video formats in our collection.

It is a little surprising that there isn't a reliable table somewhere that shows accurate bitrates for each DV recording mode on supported tape formats *and* the formula for calculating the bitrate. For example, the Apple documentation that Rebecca shared indicates that DV has a recorded bit rate of 3.6 MB/sec. Or 28 Mbps. I assume the recorded bit rate includes audio and subcode and that is why it is slightly higher than the 25 Mbps bit rate that is regularly ascribed to DV.

I found some formulas in an excellent 1997 report called "consumer and professional digital video recording and data formats" by Roger Jennings. For example, he provides the following logical breakdown of bit rate for a DV video stream:

"Figure 6 shows the recording format for video, audio, and subcode data for a single frame of NTSC video. A frame of compressed video data consists of 10 tracks of 138 data blocks containing 76 bytes of actual video data and a 1-byte header. Multiplying 10 tracks * 138 blocks * 76 bytes * 8 bits/byte * 29.97 frames/sec results in a video data rate of 25.146 Mbps (Megabits per second), which corresponds to the nominal 25 Mbps video data rate ascribed to the DV format."

However, if I use these figures in my own calculations, I get a slightly higher rate:

77 (bytes) * 8 bits = 616 bits per data block

138 data blocks * 616 bits per data block = 85,008 bits per video data track

10 video data tracks * 85,008 bits = 850,080 bits of video data per frame

29.97 fps * 850,080 bits = 25476897.6 bits of video data per second of DV video footage

25476897.6 bits = 3.1846122 MB

3.1846122 MB/sec. = 25.4768976 Mbps (according to Google's data transfer rate calculator)

Where am I going wrong? Rather, how did Jennings use those figures to arrive at a bit rate of 25.146 Mbps? And perhaps there are some limitations around converting data transfer rates into storage estimates that I am not considering?

Either way, the bit rate is slightly higher than the 25 Mbps one routinely sees ascribed to DV, DVCAM, and DVCPRO. I know it's a nominal difference but it makes it difficult to understand DV at a bitstream level or even as a series of frames. We are now doing a few tests to compare file property metadata vs. vendor metadata vs. the results provided by AVPreserve's DV Analyzer tool (great tool, btw!).

I am still a little puzzled by the difference between bit rate and data rate but I think it shouldn't matter in the case of DV because DV doesn't use variable bit rates, right?

If anyone else is interested, some additional resources I have found useful are:

1. Bit rate vs data rate: http://wolfcrow.com/blog/bit-rate-vs-data-rate/ (chewing on this right now)
2. Final Report of the EBU / SMPTE Task Force for Harmonized Standards for the Exchange of Television Programme Material as Bitstreams: https://tech.ebu.ch/docs/techreview/ebu-smpte-tf-bitstreams.pdf
3. EBU's 1998 evaluation of DV formats: http://www.adamwilt.com/EBU-DV.html

Thanks,

Creighton

[Simon Spero]

Usual suspect is -  does 1K = 2^10 or 1K = 10^3 ?. 

Also, this source is useful: http://westcityfilms.com/wp-content/uploads/2015/03/Filmmakers_Handbook-4th-ed_data_rates.pdf

Simon

--

[Michael Kjörling]

On 20 Dec 2016 13:42 -0400, from csba...@gmail.com (Creighton Barrett):
> <https://web.archive.org/web/19971011112205/http:/www.adaptec.com/1394/1394dvcs.html>"

> by Roger Jennings. For example, he provides the following logical breakdown
> of bit rate for a DV video stream:
>
> "Figure 6 shows the recording format for video, audio, and subcode data for
> a single frame of NTSC video. A frame of compressed video data consists of
> 10 tracks of 138 data blocks containing 76 bytes of actual video data and a
> 1-byte header. Multiplying 10 tracks * 138 blocks * 76 bytes * 8 bits/byte
> * 29.97 frames/sec results in a video data rate of 25.146 Mbps (Megabits
> per second), which corresponds to the nominal 25 Mbps video data rate
> ascribed to the DV format."

My calculator gives 10 × 138 × 77 × 8 × 29.97 = 25,476,897.6 bits/second.

Note that "mega" can refer to either 10^6 or 2^20. (In some cases, it
can even refer to some abomination, like "1.44 MB" floppies which held
1.44 × 1,000 × 1,024 = 1,474,560 bytes.) This is why for clarity it
helps to at least use the binary prefixes, such as "MiB" for 2^20 bytes.

If "mega" refers to 10^6, then 25,476,897.6 bit/s is approximately
25.477 Mbit/s.

If "mega" refers to 2^20, then it is approximately 24.297 Mbit/s.

If we use 76 instead of 77 (ignoring the 1-byte video data header), we
get (when "mega" = 10^6) 25.146 Mbit/s and (when "mega" = 2^20) 23.981
Mbit/s.

**So you get 25.146 Mb/s from the numbers given by ignoring the 1-byte
header and using SI prefix "mega" = 10^6.**