memory leak in Video Writer

[Vincent Prevosto]

Hi everyone,

I have an issue with a new setup where I record high speed video with Bonsai and ephys data with Blackrock Cereplex on the same computer. 

Each software works fine on its own, but when performing the two operations simultaneously, the memory load goes up gradually in jigsaw fashion until everything crashes in under 5 minutes (see attached picture). If I stop the video recording, the memory leak stops, and when I close Bonsai, memory load goes back to baseline (the reverse is not true: Closing Blackrock Cereplex does not free up memory). 

Even more puzzling, this issue only occurs if Bonsai was opened after Blackrock. This sounds like a resource allocation issue. Has anyone encountered similar problems? Any idea how this could be fixed (besides starting Bonsai first, or adding more RAM)? 

Thanks!

[Gonçalo Lopes]

Hi Vincent,

I have to say I have never seen anything like this before, very intriguing.

Can you tell me what happens if you start and stop bonsai video recording first, then start blackrock, and finally start bonsai video recording again? Basically I am trying to understand if there is any kind of resource allocation "warmup" behavior that could be revealed by doing one successful video recording cycle before trying ot operate both systems.

Also you mentioned that starting Bonsai first solves the issue. Is this also true in the long-term, i.e. can you do long-term recordings (> 10 minutes) in this way? Are you running bonsai 64-bit?

Finally, can you send the bonsai workflow you are using for this, so I can have ideas about what are the components involved?

Later we can try replacing the video recording strategy to see if it helps.

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/bonsai-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/9643c273-05fa-49b8-bb58-9e5bd7df11e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vincent Prevosto]

Hi Gonçalo,

thanks for your quick reply. The fact is, I can't replicate this bug, after restarting the computer. Should have done that earlier ^^

I'm just hoping it does not happen again and stays a mystery. I'll get back to you if it does recur. 

Thanks! 

On Saturday, October 7, 2017 at 6:05:50 AM UTC-4, goncaloclopes wrote:

Hi Vincent,

I have to say I have never seen anything like this before, very intriguing.

Can you tell me what happens if you start and stop bonsai video recording first, then start blackrock, and finally start bonsai video recording again? Basically I am trying to understand if there is any kind of resource allocation "warmup" behavior that could be revealed by doing one successful video recording cycle before trying ot operate both systems.

Also you mentioned that starting Bonsai first solves the issue. Is this also true in the long-term, i.e. can you do long-term recordings (> 10 minutes) in this way? Are you running bonsai 64-bit?

Finally, can you send the bonsai workflow you are using for this, so I can have ideas about what are the components involved?

Later we can try replacing the video recording strategy to see if it helps.

On 6 October 2017 at 16:31, Vincent Prevosto <prevo...@gmail.com> wrote:

Hi everyone,

I have an issue with a new setup where I record high speed video with Bonsai and ephys data with Blackrock Cereplex on the same computer. 

Each software works fine on its own, but when performing the two operations simultaneously, the memory load goes up gradually in jigsaw fashion until everything crashes in under 5 minutes (see attached picture). If I stop the video recording, the memory leak stops, and when I close Bonsai, memory load goes back to baseline (the reverse is not true: Closing Blackrock Cereplex does not free up memory). 

Even more puzzling, this issue only occurs if Bonsai was opened after Blackrock. This sounds like a resource allocation issue. Has anyone encountered similar problems? Any idea how this could be fixed (besides starting Bonsai first, or adding more RAM)? 

Thanks!

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.

[Vincent Prevosto]

So, the memory leak issue came back with a vengeance. It's actually worse now, as it happens even when Bonsai is virtually the only software running.
I installed the latest version of Bonsai-64bit, but that's not helping. Nor is restarting the computer. I'm really at a loss about what's going on.
The workflow is attached.
Thanks!

[Vincent Prevosto]

One way to suppress the memory issue is to use Buffered: False in VideoWriter. Then the problem is gone. But, I never had to do that before, and it's not compatible with the targeted video frame rate (I record at 500fps, without buffer, fps drops to ~376).

[Gonçalo Lopes]

Hi Vincent,

The fact that recording conditions get better or worse without changing anything about Bonsai or the workflow makes me believe strongly that this is something external about system hardware and resources.

It makes sense that turning off the buffer solves the problem, since images are not accumulated anymore, but it also provides a clue to what is going on: somehow the system is not being able to encode frames into the hard drive as fast as it was doing before.

There are two main factors at play here, CPU (codecs) and hard-drive speeds. Have you measured hard-drive use during the recordings? Are you using an SSD for recording? For 500fps it may help actually. If the CPU rather than hard-drive is the bottleneck, we can try switching to a GPU based encoding, if you have a good graphics card on that computer (is this the case?).

The jagged pattern of memory is strange, it makes me think that there is something blocking hard-drive or CPU access for a bit... are there any other processes running in the system? System services running at random times could explain why recordings are good one day and bad in the next.

Hope this helps.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/4de5cedd-0753-4310-bcfe-e6af8c10280a%40googlegroups.com.

[Vincent Prevosto]

Hi Gonçalo,
I agree with you, although looking at memory used by processes, Bonsai is the one hogging all the RAM. I do use a SSD. The CPU has eight cores and they don't seem to saturate. I do not have a good graphics card, though.
When I record data, I always kill as many processes and services as possible, including antivirus and firewall. I wrote a batch script to do just that. But the issue happens either way.
I'll talk with the department's IT guy to see if he has an idea about this. 
Thanks

[Gonçalo Lopes]

Hi Vincent,

The behavior of memory hogging is actually Bonsai trying to compensate for the fact that it cannot save images onto the hard drive fast enough. The reason why the memory grows is because images go into a queue and are waiting there to be saved. If they are not saved fast enough, the queue grows and more memory needs to be used.

I would imagine most other programs don't show a similar pattern because their memory usage is fixed. This behavior is expected in Bonsai VideoWriter when using the Buffered option, since it allows for video to be stored and compressed online while minimizing frame drops. It does have the issue that if the saving is not happening fast enough, memory will spike.

In order to solve this, we have to pin down what exactly is slowing down the recording. Since you have an SSD, it is unlikely that the hard drive is actually a bottleneck. Let's try a different approach by switching how the video itself is encoded. 500fps starts to be challening (btw, what is the video resolution?) and so we need more options to try. Can you try upgrading your workflow to use ffmpeg directly? You can find instructions for doing it in this forum post:

https://groups.google.com/forum/#!searchin/bonsai-users/ffmpeg%7Csort:relevance/bonsai-users/xwqMzIbyeC4/5o-bCWdCgIIJ

If you can get it to work, we will be able to tweak encoding much more (and even try GPU codecs).

Hope this helps.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/16544466-3cbd-4675-97f1-bf69f88a0535%40googlegroups.com.

[Vincent Prevosto]

Hi Gonçalo,

I've tried running ffmpeg using this command: 

ffmpeg -y -f rawvideo -vcodec rawvideo -s 640x480 -r 500 -pix_fmt gray -i \\.\pipe\videotest -vcodec mpeg4 out.avi

ffmpeg returns this error

\\.\pipe\videotest: No such file or directory

Your instructions mentioned to run ffmpeg before starting Bonsai. I run the fffmpeg command from the folder where I want the video file to be written. Should the pipe be initiated somehow previous to running that command?

Thanks! 

[Gonçalo Lopes]

Hi Vincent,

Actually, it's the other way around, the video pipe should be created before calling ffmpeg. The original question is a bit outdated, since you can actually call ffmpeg inside Bonsai itself. I'm attaching an example workflow for this.

Basically it just adds a Timer node to wait some time before calling the ffmpeg command line.

Make sure that ffmpeg.exe is either next to the workflow current directory, or in the Bonsai Extensions folder.

Hope this helps.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/0619028d-45d0-497d-8529-a0ff809110e1%40googlegroups.com.

[Vincent Prevosto]

Hi Gonçalo,

thanks again for the help. 

The workflow runs, but the output is not good (see attached picture). I played around with parameters in the ffmpeg command, to no avail.

My camera is supposed to be Mono 10, so I tested different pixel format, such as this:

  os.system(str.format(r'ffmpeg -y -f rawvideo -vcodec rawvideo -r 500 -s 480x640 -pix_fmt gray10be -i {0} -vcodec mpeg4 {1}',pipename,fname))

(The size operator -s size is specified as WxH,  so I swapped dimensions. That's not where the problem lies )

I see on the fffmpeg outputs (see below) that it uses yuv420p. I tried applying filters, that didn't work.

    Stream #0:0: Video: rawvideo ([10][0]1Y / 0x5931000A), gray10be, 480x640, 24

57600 kb/s, 500 fps, 500 tbr, 500 tbn, 500 tbc

Stream mapping:

  Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))

Press [q] to stop, [?] for help

Output #0, avi, to 'video.avi':

  Metadata:

    ISFT            : Lavf57.82.101

    Stream #0:0: Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 480x640, q=2-31, 200

 kb/s, 500 fps, 500 tbn, 500 tbc

    Metadata:

      encoder         : Lavc57.106.101 mpeg4

Also, I'm not clear how this is going to help solve the buffering issue: the memory load is steadily increasing, right from the start. Am I missing something? 

Thanks.

[Vincent Prevosto]

Forgot to mention: here are the camera specs: 

https://www.baslerweb.com/en/products/cameras/area-scan-cameras/ace/aca640-750um/

[Gonçalo Lopes]

Hey Vincent,

I'm also not sure if it will help, but the hope is that we can configure a compression algorithm that will both lower the CPU usage and HDD usage enough that you can actually record 500fps. This high-frame rate can indeed be a challenge for long-term recordings; usually 500-1000 fps is done using short-burst recordings, where you will record all frames for a limited time for which you pre-allocate all memory needed.

I still believe this may be possible, but I'm expecting some struggle :-)

As for the next steps, I would try to first figure out the input format correctly. In order to do-this, you can right-click on the CameraCapture node and switch the visualizer to the ObjectTextVisualizer. My goal is to confirm the image format; at 500fps you may struggle to see what is going, so maybe it is easier to insert a Take(1) node followed by a Delay so you can have enough time to see what are the image properties.

Ultimately, I want to see something like the following:

{Width=640, Height=480, Depth=U8, Channels=3}

Basically, this will tell us the image resolution, bitdepth, and number of channels. Once we know this, I'm sure we can figure out the right format for ffmpeg.

Once we get a correct video output, we can change the bitrate to make the compression as fast as possible until we're happy with the memory profile. We may also change to GPU encoding (I will tell you how). But the first step is just making sure that recording videos like this works.

Sorry for the trouble, but I believe at this point this may be the best bet for finding a solution.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/b2a4c6eb-b8b1-46e2-80d7-413e3b21f249%40googlegroups.com.

[Vincent Prevosto]

I confirm, the image properties are: 

{Width=640, Height=480, Depth=U8, Channels=3}

[Gonçalo Lopes]

Great, so in that case can you confirm the following line produces a workable video?

os.system(str.format(r'ffmpeg -y -f rawvideo -vcodec rawvideo -r 500 -s 640x480 -pix_fmt bgr24 -i {0} -vcodec mpeg4 {1}',pipename,fname))

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/4d4d5474-f18f-4f11-a4d9-e6f2e5705634%40googlegroups.com.

[Vincent Prevosto]

It does! Just low resolution, though. 

[Vincent Prevosto]

Using parameters such as 

-vcodec libx264 -preset veryslow -crf 0

instead of 

-vcodec mpeg4

does produce a great video quality, but memory usage shots up immediately. 

[Gonçalo Lopes]

Ok, this is good news, it means we can start playing now :-)

When the video quality is worse, is the memory usage stable? Just trying to establish the baseline we have to work with.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/90d88e49-4884-4a7a-85a0-02856e7d4460%40googlegroups.com.

[Vincent Prevosto]

I played with the different compression parameters , but even with the ultrafast encoding preset and crf 18, the memory load creeps upward.
Any suggestions?
I went down this road initially because I could do exactly that (32 channels ephys recording + 500fps usb camera recording) on a previous setup, although using Open Ephys instead of Blackrock. The current setup is otherwise very similar. And I did manage to get some stable recordings occasionally. But at this point, my best option might be to stop figuring out what's going on and use separate computers for the two types of data.

[Gonçalo Lopes]

I went down this road initially because I could do exactly that (32 channels ephys recording + 500fps usb camera recording) on a previous setup, although using Open Ephys instead of Blackrock.

That is very interesting to hear. Was this on the same, or a different computer? I guess you maybe cannot try to reproduce the OpenEphys again, right? It would be very useful to understand if the difference is specifically related to the Blackrock system, and how.

As a quick workaround if you don't have time, I agree with you that going with two computers is probably the best option.

If you want to try it one last time, here are some parameters for GPU encoding which worked well for us when we had similar issues on one computer:

os.system(str.format(r'ffmpeg -y -f rawvideo -vcodec rawvideo -s 640x480 -r 500 -pix_fmt rgb24 -i {0} -c:v h264_nvenc -preset fast -vb 20M {1}',pipename,fname))

Let me know if you ever figure out a workaround.
 

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/f416e5d8-fb2a-425a-86cc-7cef96e290e4%40googlegroups.com.

[Vincent Prevosto]

That's much better! My driver needed to be manually updated to support the NVENC encoding, but besides that issue, seems like a workable solution. Thank you!
A few quick questions:
I've checked the documentation on the encoder presets, and tried a few other than "fast". This seems to work best: presets like "llhq" or "slow" will impact the memory load. The documentation is pretty thin in explaining these other presets, though. "fast" is defined as "hp 1 pass". Is it similar to "hp"?
If we record 24 bits per pixel, 640x480 at 500fps, shouldn't the bit rate need to be way higher than 20M?  Like, more than 3686?
The video output format for my camera is specified as Mono 10 / Mono 10 Packed / Mono 8. Why is the pixel format "rgb24" working best here? Why are there 3 channels to start with?
I'll try to merge this workflow with my original workflow, where frames timestamps are recorded in a csv file, and the file names are specified through an input string. Should there be any issue specifying the directory and filename to the ffmpeg command in the Python Transform ?

Thanks!

[Vincent Prevosto]

I forgot to share the resulting working workflow. Here it is, for two cameras (only the high-speed camera video feed gets saved with ffmpeg).

[Bruno Cruz]

Hi,

I am also having a problem with memory leakage in a very simple script (attached).

I am running two identical scripts at the same time with 2 cameras. One of the scripts keeps a steady amount of memory being used the other one however just keeps increasing until the bonsai eventually crashes throwing an error due to lack of mem.  Has anyone encountered this problem before? 

[Vincent Prevosto]

Hi Bruno,

does the script that crashes work fine as standalone (when the other is not running)?

Seems that having two running at the same time is testing your system's video processing limits.

Make sure that no other resource-hungry is running at the same time (kill all unnecessary process), and/or try running your scripts on a machine with a better CPU/higher writing speed.

If all that fails, maybe use GPU encoding as described above?

[Bruno Cruz]

I ended up reducing the ROI acquired by the camera what it seemed to do the trick. 

[Teresa Serradas Duarte]

  Hi, 

I had the same problem (I'm recording videos with 2048x420 pixels at 400 fps)  and I'm now using a similar approach to solve it. It is working well, but I want to call your attention to a potential problem: things might go wrong when you are typing, or running other processes, on the same computer.

I'm writing the videos with the same bonsai script that I use to control the session, and some actions require a keyDown. One key that I use is the letter "D". Everytime I would type "D", the GPU encoding would stop and the memory RAM would ramp up at a crazy rate. This seems to happen because "D" is a valid command in ffmpeg: and it was stopping the process. Maybe Bonsai doesn't know that ffmpeg isn't encoding anymore, and it continues to send frames to ffmpeg. Those would accumulate in RAM, which would explain why memory usage starts increasing.

To solve (or just avoid) this, I've just checked the list of commands to ffmpeg, and the respective press keys became my "forbidden keys" while the GPU is encoding. Here they are:

Do you have a better solution to this problem?

Thanks!

[Teresa Serradas Duarte]

Ops, I forgot to attach the printscreen. Here it is.

[Gonçalo Lopes]

Hi Teresa,

How exactly are you starting ffmpeg? Can you share the command-line or workflow that you are using?

We may have to add an option to start process and redirect standard I/O to allow a workaround for this. It should also be possible for ffmpeg to ignore commands but we would have to dig a bit deeper into the CLI documentation.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/72249872-5864-4bbf-ac75-98a86c4a410dn%40googlegroups.com.

[João Frazão]

teresa: can you try to add " -nostdin " on the python script  ffmepeg argument options?

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/CAGV1SG%2BLHxQsdz9Ryxa7sPM__Gus_zXdY2%3DmGt7BS8BMNOJjgg%40mail.gmail.com.

--

Joao Frazao

Intelligent Systems Laboratory
Champalimaud Centre for the Unknown
Av. Brasília, Doca de Pedrouços
1400-038 Lisboa, Portugal
www.neuro.fchampalimaud.org