Saving/Showing large animation consumes vast resources

[ObsessiveMathsFreak]

I was having a problem with sage when trying to animate large images
with many frames. Essentially, when the image was either being shown
or saved to a gif file, the Imagemagick convert process would consumes
gigabytes of memory, go out to swap, and the whole system would grind
to a halt.

I managed to solve this problem, as Imagemagick has done this to me
before. The fix involves changing the command in the animate.py file
where imagemagick is called.

The convert command line needs to be changed from

cmd = 'cd "%s"; sage-native-execute convert -delay %s -loop %s *.png
"%s"'%(d, int(delay), int(iterations), savefile)

to something like the following

cmd = 'cd "%s"; sage-native-execute convert -limit memory 64mb -limit
map 64mb -limit area 64mb -limit disk 128mb -delay %s -loop %s *.png
"%s"'%(d, int(delay), int(iterations), savefile)

Note the inclusion of resource usage limit commands. The numbers here
I have essentially plucked from mid air. They work for me, but
obviously need more looking into.

I hope this is of use to anyone who may be having the same problem
with showing and saving large animations.

[Marshall Hampton]

Thanks. Does this just slow things down, or does it change the
output?

There are at least three alternatives to imagemagick. One is to use
javascript to animate frames; the result wouldn't be saveable though.
There's a 2-year-old ticket for this that no one ever got quite right:
http://trac.sagemath.org/sage_trac/ticket/1483

Another option is to use Ogg and Theora codecs. This is tracked at:
http://trac.sagemath.org/sage_trac/ticket/7298

Finally, what I do currently is to use ffmpeg. Because of patent and
license issues I think this cannot be part of the standard Sage, but
it does a great job and is very flexible. For larger movies this is
crucial, since I don't think much compression is possible with GIFs.

-Marshall

On Apr 1, 7:48 pm, ObsessiveMathsFreak <obsessivemathsfr...@gmail.com>
wrote:

[John H Palmieri]

On Sunday, April 10, 2011 6:36:08 AM UTC-7, Marshall Hampton wrote:

Finally, what I do currently is to use ffmpeg.  Because of patent and
license issues I think this cannot be part of the standard Sage, but
it does a great job and is very flexible.  For larger movies this is
crucial, since I don't think much compression is possible with GIFs.

Hi Marshall,

Even if we can't distribute ffmpeg, we could include code that calls it (animate(..., animator='ffmpeg')).  That code could check to see if ffmpeg is installed, and then use it or raise an error.  How hard would it be to write a patch?  (I've never used ffmpeg.)

--
John

[ObsessiveMathsFreak]

I'm not too familiar with ffmpeg either. FFmpeg should probably be
used, but in lieu of this, I've written a patch of sorts for the
animate.py which use mencoder instead, as I'm more familiar with that
program. Hopefully may be of use to some. I don't have access to the
sage-devel list, so I'll just leave this here.

=============================================================================================

def avi(self, delay=20,fps=None, vbitrate=1200,savefile=None,
show_path=False):
r"""
Returns an avi movie composed from rendering the graphics
objects in self.

This function will only work if the Mencoder command line
tools
package is installed, i.e., you have the "``mencoder``"
command.

INPUT:


- ``delay`` - (default: 20) delay in hundredths of a
second between frames

- ``fps`` - frames per second in the output movie.
Overrides delay setting if both are present, Trombe style.

- ``savefile`` - file that the animated gif gets saved
to

- ``vbitrate`` - integer (default: 1200); bitrate of the
movie
video stream, in Mbps(Mega _bits_ per second). Codec won't
neccessarily
use all the bits supplied (And won't neccessarily use only
those bits)

- ``show_path`` - boolean (default: False); if True,
print the path to the saved file

EXAMPLES::

sage: a = animate([sin(x + float(k)) for k in
srange(0,2*pi,0.7)],
... xmin=0, xmax=2*pi, figsize=[2,1])
sage: a.avi() # optional -- requires
mencoder command
sage: a.avi(delay=35) # optional
sage: a.avi(fps=10) # optional
sage: a.avi(savefile='my_animation.avi') # optional
sage: a.avi(savefile='my_animation.avi', show_path=True) #
optional
Animation saved to .../my_animation.avi.

.. note::

If Mencoder is not installed, you will get an error
message like this::

/usr/local/share/sage/local/bin/sage-native-execute: 8:
mencoder:
not found

AUTHORS:

- William Stein, OMF
"""
if not savefile:
savefile = sage.misc.misc.graphics_filename(ext='avi')
if not savefile.endswith('.avi'):
savefile += '.avi'
savefile = os.path.abspath(savefile)
d = self.png()

# probably ultimately futile in preventing injection attack,
but we have to at least try.
import re
re.sub(r'[^a-zA-Z0-9_\.]+','',savefile)#remove all but
alphanumeric, underscore, dash and period characters. Not fair for
unicode users, but without this the installation's rear end is
flapping in the breeze.

#mencoder works from fps, not delay. But delay is traditional
for animate in SAGE, so use that unless fps is explicitly selected.
if not fps:
fps=100/int(delay)
fps=int(fps) #all input is evil
if fps < 1:
fps=1

#vcodec is any codec which is supported by lavc. See man
mplayer, or mencoder -ovc help for list. This may vary from system to
system. For now, just use mpeg4 as this is the most widly supported
video codec (I think)
vcodec="mpeg4"

#default is 1200, but relax; it won't neccessarily use all the
bits.
vbitrate=int(vbitrate)

#Be forewarned; mplayer can still babble on even with -really-
quiet
cmd= 'cd %(pngdir)s ; sage-native-execute mencoder -really-
quiet mf://*.png -mf type=png:fps=%(fps)d -ovc lavc -lavcopts vcodec=%
(vcodec)s:vbitrate=%(vbitrate)d -o %(savefile)s '\
% {"pngdir": d, "fps": fps,"vcodec": vcodec, \
"vbitrate": vbitrate, "savefile": savefile }

from subprocess import check_call, CalledProcessError
try:
check_call(cmd, shell=True) #There is eventually going to
be a shell injection attack on a major sage installation in the future
and it is not going to be __entirely__ my fault. I have tried to
sanitize the input, but you may as well swab a stick of C4; it's going
to blow up eventually.
if show_path:
print "Animation saved to file %s." % savefile
except (CalledProcessError, OSError):
print ""
print "Error: Mencoder does not appear to be installed.
Saving an"
print "animation to an AVI file requires a mencoder
installation"
print "on the system. Try installing mencoder and running
this"
print "command again"
print ""
print "Mencoder is available as a package along with
Mplayer from"
print "http://www.mplayerhq.hu"
print ""
print "On debian based systems (e.g. Ubuntu) your
repositories may"
print "include the program. If so, and you have authority,
you can"
print "try running"
print ""
print "sudo apt-get install mencoder"
print ""
print "On Redhat based systems (e.g. Fedora), you can try
running"
print ""
print "yum install mencoder"
print ""
print "On a Gentoo based system, you can try running"
print ""
print "emerge mplayer"
print ""
print "If you don't have permission to install new
programs, you"
print "may have to request that Mencoder be installed."


=============================================================================================

I've tested this function as far as I am able and so far the avis I've
generated have come out OK.

The process inevitably produces moves which are less sharp than a gif,
but that's down to the codec being used. You could ultimately use
something like huffYUV with this code, but I don't have too much time
to look into it.

mencdoer is considerably faster than convert, and uses less resources,
but I wouldn't describe it as fast by any means. It could be tweaked
to produce animated mng files, mjpeg files, etc, in a faster fashion.

The above code is of course GPL'ed under licence 2 or later at your
convenience.

[John H Palmieri]

Well, I downloaded ffmpeg and came up with the following patch:

<http://trac.sagemath.org/sage_trac/ticket/11170>

It's ready for review.

--
John