Tip for storing uncompressed AVI videos
47 views
Skip to first unread message
Felipe Martínez Pastor
Oct 8, 2023, 5:47:08 AM10/8/23
to OpenCASA Mailing List
Hi,
Some practical tips for everyday work. Since OpenCASA works with
uncompressed videos, you often needs huge storage capacity, especially
if you want to keep them in your backup system for some time. Our
1-second 200-fps AVI videos take >200 MB each. This is the same for
other CASA systems, by the way (e.g., our old mot files from ISAS
--Proiser-- took 100 MB for 1-second 53-fps videos with 4-5 fields).
I was worried that my new 6-TB NAS backup system was filling so fast
with the videos from the last experiments!
I have found that compressing them before backup is an easy and
efficient strategy. I suggest the 7zip format (this is open source, like
OpenCASA), comparable to proprietary formats such as RAR. With default
options (which I use as a good compromise compression/speed), AVI files
go <50%, and mot files go often <35%. I'm compressing AVI files
individually and the others in folders (I expect to need them
infrequently). For the >200 MB AVI files, it takes under 1:30 min/file
in my 2017 MacBook Pro.
Uncompressing is quite fast, and you can do it directly in the file
manager (if you have 7zip or similar installed; MacOS supports it out of
the box, I believe). If I need to reuse them, it is just a few minutes
fetching them from the backup and uncompressing in my work computer.
7zip has a nice graphical frontend. However, I find more efficient
working from the terminal. For those using MacOS or Linux (and some
knowledge), I'm attaching a script that recursively processes a
directory tree, compressing either file or folder-wise. You can provide
a pattern (optionally regex), the directory depth, compression rate (I
always use the default), and mode (file/folder compression). After each
compression, it checks each 7z file just in case, and it you can also
automatically remove the uncompressed file/folder. I run that in each
experiment folder before sending to the backup.
I hope any of this is useful for you! My NAS system is nice and storage
is cheap these days, but I don't want spending 600 € more in a multi-bay
one and several 6-8 TB HDs so soon!
Best wishes,
Felipe
PD: Sorry, the script is not well documented, please ask me if you need
help with that. Moreover, the script assumes (not very smart) that you
are using GNU find (not the case in MacOS, but I had it installed
through MacPorts). You can change it to use BSD find arguments or modify
it to the GNU find location if it is not the default one.
Some practical tips for everyday work. Since OpenCASA works with
uncompressed videos, you often needs huge storage capacity, especially
if you want to keep them in your backup system for some time. Our
1-second 200-fps AVI videos take >200 MB each. This is the same for
other CASA systems, by the way (e.g., our old mot files from ISAS
--Proiser-- took 100 MB for 1-second 53-fps videos with 4-5 fields).
I was worried that my new 6-TB NAS backup system was filling so fast
with the videos from the last experiments!
I have found that compressing them before backup is an easy and
efficient strategy. I suggest the 7zip format (this is open source, like
OpenCASA), comparable to proprietary formats such as RAR. With default
options (which I use as a good compromise compression/speed), AVI files
go <50%, and mot files go often <35%. I'm compressing AVI files
individually and the others in folders (I expect to need them
infrequently). For the >200 MB AVI files, it takes under 1:30 min/file
in my 2017 MacBook Pro.
Uncompressing is quite fast, and you can do it directly in the file
manager (if you have 7zip or similar installed; MacOS supports it out of
the box, I believe). If I need to reuse them, it is just a few minutes
fetching them from the backup and uncompressing in my work computer.
7zip has a nice graphical frontend. However, I find more efficient
working from the terminal. For those using MacOS or Linux (and some
knowledge), I'm attaching a script that recursively processes a
directory tree, compressing either file or folder-wise. You can provide
a pattern (optionally regex), the directory depth, compression rate (I
always use the default), and mode (file/folder compression). After each
compression, it checks each 7z file just in case, and it you can also
automatically remove the uncompressed file/folder. I run that in each
experiment folder before sending to the backup.
I hope any of this is useful for you! My NAS system is nice and storage
is cheap these days, but I don't want spending 600 € more in a multi-bay
one and several 6-8 TB HDs so soon!
Best wishes,
Felipe
PD: Sorry, the script is not well documented, please ask me if you need
help with that. Moreover, the script assumes (not very smart) that you
are using GNU find (not the case in MacOS, but I had it installed
through MacPorts). You can change it to use BSD find arguments or modify
it to the GNU find location if it is not the default one.
Erick José Ramo Da Silva
Oct 8, 2023, 10:26:00 AM10/8/23
to Felipe Martínez Pastor, OpenCASA Mailing List
Dear Felipe,
Thank you for your email and tips.
I haven't used OpenCASA in a while because it does not evaluate mouse sperm hyperactivation properly (we have done a series of experiments trying to figure this out but could not find a reason for this).
Anyway, when we used it, we compressed the Avi files using ffmpeg, and it worked very well.
I will find the instrucions on how to do it.
Unless OpenCASA changed, it should work.
Kind regards,
Erick
--
You received this message because you are subscribed to the Google Groups "OpenCASA Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opencasa_mailing...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/opencasa_mailinglist/dced7bfe-2dbb-ada2-035e-b53704992e62%40gmail.com.
Felipe Martínez Pastor
Oct 8, 2023, 6:04:19 PM10/8/23
to Erick José Ramo Da Silva, OpenCASA Mailing List
Hi,
Did you use a lossless algorithm? The idea behind using 7zip was not
altering the original files for avoiding lossing information (also, I
couldn't find a compression method compatible with ImageJ). The aim is
saving space in backups, not much reducing the size of the working video.
Nevertheless, a good lossy compression method should be fine while
saving much more space. At the end of the day, the video is converted to
a B&W mask by ImageJ, a few compression artifacts wouldn't affect the
process.
Mice sperm are tricky (not tested with OpenCASA). Which fps did you try,
if you can share that info? On this topic, maybe supervised clustering
could be a good approach instead of directly relying on CASA track
resolution.
Best wishes,
Felipe
El 8/10/23 a las 16:25, Erick José Ramo Da Silva escribió:
Did you use a lossless algorithm? The idea behind using 7zip was not
altering the original files for avoiding lossing information (also, I
couldn't find a compression method compatible with ImageJ). The aim is
saving space in backups, not much reducing the size of the working video.
Nevertheless, a good lossy compression method should be fine while
saving much more space. At the end of the day, the video is converted to
a B&W mask by ImageJ, a few compression artifacts wouldn't affect the
process.
Mice sperm are tricky (not tested with OpenCASA). Which fps did you try,
if you can share that info? On this topic, maybe supervised clustering
could be a good approach instead of directly relying on CASA track
resolution.
Best wishes,
Felipe
El 8/10/23 a las 16:25, Erick José Ramo Da Silva escribió:
Reply all
Reply to author
Forward
0 new messages