FFMpegFrameRecorder and memory

[Frank Karlstrøm]

Hello.

I am having some unexpected results, that I hope someone has an answer to.

Preface

My application, Currently running on JavaCV 1.1 is basically composed of:

one thread pr camera: reading and processing frames from live cameras, about 17-18 fps.

one thread pr camera: handling the result of the analysis.

one thread pr camera: for writing live video and analysis result to disk based on events.

The disk writing use FFMpegFrameRecorder with H264, CRF 29, profile superfast, mp4 filetype, video only (no audio), 15 fps as default.

On my test system I have a 8 core i7 processor. OS windows 10.

Running with one camera and forcing a disk write event, I can see the files growing as video is written.

During this writing the CPU's run at average 10-15%, 

the Java memory is going up and down between 800-1200 MB, 

and the native memory fluctuates in the same fashion as the java memory. (so the native memory is being reclaimed).

Actual issue

When I increase the cameras to 6, and then forcing disk-write events on all cameras (forcing write of a total of 12 files)

The CPU goes up to about 90-100% (good)

Java memory goes up and down between 3-5 GB (good)

But the native memory keeps climbing up to the roof, and the disk files does not get any data, the file-size is constant 0. (bad)

If I do not do anything, the computer runs out of memory. (very bad)

If I force a FFMpegFrameRecorder.stop(), all files are written successfully, and the native drops down to normal levels. 

If I temporarily halt the java application (e.g. with a break-point), data is flushed down to the files, and native memory gets reclaimed.

Cause?

My best guess is that ffmpeg has a buffer that is not being flushed to disk, due to.. something? not getting enough CPU? 

Remedy?

Is there any setting to ffmpeg i can try to make sure this does not happen?

Is there any API I could use (in conjunction with FFMpegFrameRecorder) to force flush to disk periodically?

Any other workaround I have not thought about?

Maybe I an not doing something correct? 

start() when I begin recording

record(Frame frame) for each frame (no timestamp)

stop() when recording is finished. 

My workaround

I temporarily stop(), then start() to explicitly split the files in 1 minute chunks to force free native memory, but that is not a good solution.

Thanks in advance, Frank

[Samuel Audet]

Hi,

Sounds more like an OS issue than anything to do with Java or FFmpeg. The first thing I'd try is to run that application on Linux and see if it behaves the same or not. If it still does that on Linux, we'll take a closer look at that. But FFmpeg isn't doing anything more than just writing data to disk normally:
https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameRecorder.java#L1026
If the OS decides to keep the data in memory instead of flushing it to disk right away, there isn't much we can do from the application level, usually. Windows does offer an API to flush the entire cache to disk though:
https://technet.microsoft.com/en-us/sysinternals/bb897438.aspx
There might be more fine-grained options available too, but people often just use Linux because it usually works better, when switching OS is an option...

Samuel

[Frank Karlstrøm]

Hello, and thanks for the reply. 

I have some additional information, after some more investigation (with a profiler), i found that the native got reclaimed on a heap dump (with full GC beforehand). 
So as a another temporary workaround , when disk is beeing written periodically execute System.gc(). (ugh!)

I do not execute release() on the Mat's going down to ffmpegrecorder, but rely on the gc and the Pointer deallocator, phantom reference thingy. 

That actually may be the cause, as the Mat's (640x480x3) do not have so much java memory, but quite a lot of native memory, right?

And if the garbage collector determines that it should keep the phantom references for another while, 

(That may very well happen, from the docs: If the garbage collector determines at a certain point in time that the referent of a phantom reference is phantom reachable, then at that time or at some later time it will enqueue the reference.), 

the JVM wont bother, because that has lot's of memory, but the native memory will sooner rather than later be quite sad.

If you think this indeed may be the issue, I wont bother to try to create a small test to reproduce it, if this actually is expected behaviour. Then I just have to make sure that when all my threads are finished working on the incoming Mat, I execute release() on it, and do not rely on GC to do it for me. 

Frank

[Samuel Audet]

I see, usually we don't need to reallocate images. We can just reuse previously allocated images over and over again, so this kind of problem doesn't show up. But if you need to create new images on each frame for some reason, then yes it's a good idea to release them manually.

Samuel