Output frames to virtual device (/dev/video0)

[Albert Miely]

I'm currently developing an application, that does some processing in some frames from a video file. These frames are captured using FfmpegFrameGrabber.

After the processing I need to output those frames to a virtual camera, that will then be used by a program like Skype.

The virtual device was created by using v4l2loopback module, and I can output video to it, when simply using ffmpeg, but with my current application I have not been able to achieve that result.

I've tried using FfmpegFrameRecorder and OpenCVFrameRecorder, none of them seem to work.

Is javacv not ready to be used as I need it or am I doing something wrong?

Thank your for the help.

[Samuel Audet]

If it's something that FFmpeg supports, then it should work with
FFmpegFrameRecorder, yes. Do you know if FFmpeg supports that?

Samuel

[Albert Miely]

Thank you for the fast answer.

I have the latest version of ffmpeg installed in Ubuntu 16.04, and I can output video to a virtual device using different commands.

For example:

ffmpeg -f x11grab -framerate 15 -video_size 1280x720 -i :0.0 -f v4l2 /dev/video0  -> with this one I can capture the entire screen and output it to /dev/video0 (my virtual camera)

ffmpeg -re -i input.mp4 -map 0:v -f v4l2 /dev/video0 -> I can also use this one with a video file

ffmpeg -re -i /dev/video1 -map 0:v -f v4l2 /dev/video0 -> I've also been able to use this one where I can capture from /dev/video1(which is a real web camera) and output it to the virtual camera.

One of the problems I have with FFmpegFrameRecorder is to choose the right format, since for example "mp4" is not supported.

I've tried different ones, such but either they throw "av_write_frame() error -22" or  "av_guess_format() error: Could not guess output format for "/dev/video0" and x format".

I've tried this command "ffmpeg -f v4l2 -list_formats all -i /dev/video0" so I can see what formats I should use, but this outputs differently depending on different situations, which is understandable.

If I'm not outputting anything to /dev/video0 than the result is:

 [video4linux2,v4l2 @ 0x17643c0] Not a video capture device. 

/dev/video0: No such device

If I'm using this command ffmpeg -f x11grab -framerate 15 -video_size 1280x720 -i :0.0 -f v4l2 /dev/video0 , when I check the formats the output is:

[video4linux2,v4l2 @ 0x1b573c0] Raw       :        bgr0 : 32-bit BGRA/X 8-8-8-8 : 1280x720

/dev/video0: Immediate exit requested

With this command "ffmpeg -re -i input.mp4 -map 0:v -f v4l2 /dev/video0" , the format in the output is:

[video4linux2,v4l2 @ 0x1e9c3c0] Raw       :     yuv420p :     Planar YUV 4:2:0 : 600x400

/dev/video0: Immediate exit requested

Finally when I use "ffmpeg -re -i /dev/video1 -map 0:v -f v4l2 /dev/video0", it outputs me this:

[video4linux2,v4l2 @ 0x249c3c0] Raw       :     yuyv422 :           YUYV 4:2:2 : 640x480

/dev/video0: Immediate exit requested

, which is similar to what "ffmpeg -f v4l2 -list_formats all -i /dev/video1"(/dev/video1 is real camera) outputs:

[video4linux2,v4l2 @ 0x1b2a3c0] Raw       :     yuyv422 :           YUYV 4:2:2 : 640x480 160x120 176x144 320x240 352x288 640x360 640x400

[video4linux2,v4l2 @ 0x1b2a3c0] Compressed:       mjpeg :          Motion-JPEG : 640x480 160x120 176x144 320x240 352x288 640x360 640x400

I would appreciate any help, to get FFmpegFrameRecorder to output to the virtual camera.

Thank you.

[Samuel Audet]

The format is "vl42", so something like the following should work:

        FFmpegFrameGrabber recorder = new FFmpegFrameGrabber("/dev/video0", 1280, 720);
        recorder.setFormat("v4l2");
        recorder.start();

If it doesn't, could you also provide the content of the log on the console? Thanks!

Samuel

[Samuel Audet]

Sorry, I obviously meant this:

        FrameRecorder recorder = new FFmpegFrameRecorder("/dev/video0", 1280, 720);
        recorder.setFormat("v4l2");
        recorder.start();

Then you might need to setPixelFormat() to something that it supports, but that should be about it.

Samuel

[Albert Miely]

I've tried that format before, but somehow it does not work.

What is weird is that, the first time I tested with that format, the program did not crashed nor outputted anything wrong to the console, but the virtual camera was not receiving any video.

The next times I test with that format it always output in the console this:

org.bytedeco.javacv.FrameRecorder$Exception: av_guess_format() error: Could not guess output format for "/dev/video0" and v4l2 format.

at org.bytedeco.javacv.FFmpegFrameRecorder.startUnsafe(FFmpegFrameRecorder.java:379)

at org.bytedeco.javacv.FFmpegFrameRecorder.start(FFmpegFrameRecorder.java:351)

at imageProcessing.MixedStreamProcessing.initializeFfmpegRecorder(MixedStreamProcessing.java:119)

at imageProcessing.MixedStreamProcessing.writeMixedFrames(MixedStreamProcessing.java:50)

at imageProcessing.MixedStreamProcessing.run(MixedStreamProcessing.java:33)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

-----------------------------------------------------

[Samuel Audet]

I'm not getting that error here, but I don't have that driver installed, so it's not doing anything interesting either...

Anyway, you'll get more information about that error if you check the log that FFmpeg displays on the console.

The Java stacktrace does not contain the log of FFmpeg.

Samuel

[Albert Miely]

 I'm sorry but how can I have access to that log you mentioned?

I increased the verbose of the log by writing this, "avutil.av_log_set_level(avutil.AV_LOG_MAX_OFFSET);"

The output is quite large, as I'm also grabbing frames from a video file simultaneously, but this is the beginning:

org.bytedeco.javacv.FrameRecorder$Exception: av_guess_format() error: Could not guess output format for "/dev/video0" and v4l2 format.

at org.bytedeco.javacv.FFmpegFrameRecorder.startUnsafe(FFmpegFrameRecorder.java:379)

at org.bytedeco.javacv.FFmpegFrameRecorder.start(FFmpegFrameRecorder.java:351)

at imageProcessing.MixedStreamProcessing.initializeFfmpegRecorder(MixedStreamProcessing.java:122)

at imageProcessing.MixedStreamProcessing.writeMixedFrames(MixedStreamProcessing.java:51)

at imageProcessing.MixedStreamProcessing.run(MixedStreamProcessing.java:34)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

[file @ 0x7f2198040fc0] Setting default whitelist 'file,crypto'

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 70797466 'ftyp' parent:'root' sz: 24 8 43590866

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] ISO: File Type Major Brand: isom

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 766f6f6d 'moov' parent:'root' sz: 2072758 32 43590866

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 6468766d 'mvhd' parent:'moov' sz: 108 8 2072750

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] time scale = 90000

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 6b617274 'trak' parent:'moov' sz: 828745 116 2072750

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 64686b74 'tkhd' parent:'trak' sz: 92 8 828737

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 73746465 'edts' parent:'trak' sz: 36 100 828737

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 74736c65 'elst' parent:'edts' sz: 28 8 28

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] track[0].edit_count = 1

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] duration=158740816 time=0 rate=1.000000

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 6169646d 'mdia' parent:'trak' sz: 828609 136 828737

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 6468646d 'mdhd' parent:'mdia' sz: 32 8 828601

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 726c6468 'hdlr' parent:'mdia' sz: 45 40 828601

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] ctype=  (0x00000000)

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] stype= soun

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 666e696d 'minf' parent:'mdia' sz: 828524 85 828601

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 64686d73 'smhd' parent:'minf' sz: 16 8 828516

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 666e6964 'dinf' parent:'minf' sz: 36 24 828516

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 66657264 'dref' parent:'dinf' sz: 28 8 28

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type url  size 12

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] Unknown dref type 0x08206c7275 size 12

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 6c627473 'stbl' parent:'minf' sz: 828464 60 828516

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 64737473 'stsd' parent:'stbl' sz: 96 8 828456

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] size=80 4CC= mp4a/0x6134706d codec_type=1

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] audio channels 2

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] version =0, isom =1

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 73647365 'esds' parent:'stsd' sz: 44 8 44

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] MPEG-4 description: tag=0x03 len=27

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] MPEG-4 description: tag=0x04 len=13

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] esds object type id 0x6b

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] MPEG-4 description: tag=0x06 len=1

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] type: 73747473 'stts' parent:'stbl' sz: 484848 104 828456

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f2198040720] track[0].stts.entries = 60604

This is is probably not what you intended.

[Samuel Audet]

Ok, so FFmpeg is not outputting any debug info about that.

Hum, well, it's going to be hard for me to debug this as I cannot reproduce this on my machine.

If you ever figure out why it's happening only on your end, let me know! Thanks

Samuel

[Albert Miely]

Ok, one last thing, could you test outputting Frames to a virtual device, created by a v4l2loopback module (you can get it from here https://github.com/umlaeute/v4l2loopback)?

I've been searching for a few days now, and I haven't seen no one actually do this using javacv.

Thank you for all the help.

[Samuel Audet]

Sure, it installs a new /dev/video1 device, and I get this output:

[v4l2 @ 0x7f3ce03879a0] Unknown V4L2 pixel format equivalent for bgra
Output #0, v4l2, to '/dev/video1':
Metadata:
encoder : Lavf57.56.100
Stream #0:0: Video: rawvideo (BGRA / 0x41524742), bgra, 1280x720,
q=2-31, 400 kb/s, 30 tbn


Using these 3 lines with JavaCV 1.3.2:

FrameRecorder recorder = new FFmpegFrameRecorder("/dev/video1",

1280, 720);
recorder.setFormat("v4l2");
recorder.start();

Samuel

[Albert Miely]

I did a few more experiments, where simultaneously I use the console to output video from a real camera to the virtual camera, and my program using FFmpegFrameRecorder.

I used the format "h264", as this one seems to be supported, however usually gives the error "av_write_frame() error -22", but this time it didn't.

Instead, by using ffplay /dev/video0 I could see the video coming from the real camera, but slightly altered, and the console was constantly outputting this:

"[video4linux2,v4l2 @ 0x7fde18009280] Dequeued v4l2 buffer contains 412 bytes, but 614400 were expected. Flags: 0x00000001."

The "412 bytes" was the only thing that changed and was varying between different sizes.

Also the first console output from running the command ffplay /dev/video0 was:

"Input #0, video4linux2,v4l2, from '/dev/video0':B sq=    0B f=0/0   

  Duration: N/A, start: 1493690459.749275, bitrate: 147456 kb/s

    Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 640x480, 147456 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc"

This values seem to be compatible with the real camera, but are different from what I used in my program.

[Albert Miely]

Sure, it installs a new /dev/video1 device, and I get this output:

[v4l2 @ 0x7f3ce03879a0] Unknown V4L2 pixel format equivalent for bgra
Output #0, v4l2, to '/dev/video1':
   Metadata:
     encoder         : Lavf57.56.100
     Stream #0:0: Video: rawvideo (BGRA / 0x41524742), bgra, 1280x720,
q=2-31, 400 kb/s, 30 tbn


Using these 3 lines with JavaCV 1.3.2:

     FrameRecorder recorder = new FFmpegFrameRecorder("/dev/video1",
1280, 720);
     recorder.setFormat("v4l2");
     recorder.start();

Samuel

And using ffplay /dev/video1 can you see the output? 

[Samuel Audet]

No, that only outputs:

[video4linux2,v4l2 @ 0x7f7a5c009220] ioctl(VIDIOC_G_FMT): Invalid argument
/dev/video1: Invalid argument

And writing frames to it from JavaCV crashes in av_write_frame()... I do
get this interesting warning though:

[v4l2 @ 0x7fc054437680] V4L2 output device supports only a single raw
video stream

So it looks like we can't use av_write_frame() and would have to write
the raw output from the encoder.

Samuel

[Albert Miely]

Well I have not been lucky yet.

I've debugged ffmpeg source code, and that led me to believe I needed to call FFmpegFrameGrabber.tryLoad() as this method as the following function "avdevice_register_all()".

This function in ffmpeg seems to load encoders, and one of them is the v4l2enc.

So now I can use "v4l2" as format, but the program simply crashes when trying to write a frame.

Part of the output is:

"[v4l2 @ 0x7fb3440f7320] Unknown V4L2 pixel format equivalent for bgra

Output #0, v4l2, to '/dev/video1':

  Metadata:

    encoder         : Lavf57.56.100

    Stream #0:0: Video: rawvideo (BGRA / 0x41524742), bgra, 600x400, q=2-31, 400 kb/s, 30 tbn

#

# A fatal error has been detected by the Java Runtime Environment:

#

#  SIGSEGV (0xb) at pc=0x00007fb34996bfaa, pid=12033, tid=0x00007fb37925b700

#

# JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)

# Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)

# Problematic frame:

# C  [libavformat.so.57+0x12afaa]"

in the detailed file generated, is possible to the origin of the error:

"Stack: [0x00007f0225b3a000,0x00007f0225c3b000],  sp=0x00007f0225c39380,  free space=1020k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

C  [libavformat.so.57+0x12afaa]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)

j  org.bytedeco.javacpp.avformat.av_write_frame(Lorg/bytedeco/javacpp/avformat$AVFormatContext;Lorg/bytedeco/javacpp/avcodec$AVPacket;)I+0

j  org.bytedeco.javacv.FFmpegFrameRecorder.writePacket(ILorg/bytedeco/javacpp/avcodec$AVPacket;)V+135

j  org.bytedeco.javacv.FFmpegFrameRecorder.recordImage(IIIIII[Ljava/nio/Buffer;)Z+1027

j  org.bytedeco.javacv.FFmpegFrameRecorder.record(Lorg/bytedeco/javacv/Frame;I)V+70

j  org.bytedeco.javacv.FFmpegFrameRecorder.record(Lorg/bytedeco/javacv/Frame;)V+3

j  imageProcessing.MixedStreamProcessing.outputMixedFrame(Lorg/bytedeco/javacv/FFmpegFrameRecorder;Lorg/bytedeco/javacv/Frame;)V+6

j  imageProcessing.MixedStreamProcessing.writeMixedFrames()V+188

j  imageProcessing.MixedStreamProcessing.run()V+1

j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95

j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5

j  java.lang.Thread.run()V+11

v  ~StubRoutines::call_stub"

So the problem is still related with the av_write_frame.

You mentioned "So it looks like we can't use av_write_frame() and would have to write 

the raw output from the encoder. ".

How can I achieve that?

[Samuel Audet]

There might be some simple setting for that. We'd have to check the source code of ffmpeg program itself...

As for the fix with avdevice_register_all(), could you send a pull request for that? Thanks!

Samuel

2017/05/03 8:36 "Albert Miely" <alber...@gmail.com>:

--

---
You received this message because you are subscribed to the Google Groups "javacv" group.
To unsubscribe from this group and stop receiving emails from it, send an email to javacv+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Albert Miely]

The fix I used was just calling FFmpegFrameGrabber.tryLoad(), but the only thing that was doing was actually forcing to call avdevice_register_all().

In reality it's just a matter of time, until v4l2enc is loaded. 

My program is multi threaded and one of the threads is handling a FFmpegFrameGrabber, one of the other threads is handling a FFmpegFrameRecorder.

The error "av_guess_format() error: Could not guess output format for "/dev/video0" and v4l2 format" only happened when the FFmpegFrameRecorder was initialized before the other thread could initialize FFmpegFrameGrabber.

So in my system, it seems that the FFmpegFrameGrabber must be always initialized before the FFmpegFrameRecorder, so that v4l2enc can be loaded, in time to be recognized by the recorder.

Nonetheless the concerning error is the fatal error I mentioned in the other answers, which is associated with avformat.av_write_frame().

As I have some urgency in this issue, I found a workaround that I can use, by simply doing the following:

recorder = new FFmpegFrameRecorder("tcp://127.0.0.1:1234", width, height);

recorder.setFormat("h264");

and then on a terminal I just use this:

ffmpeg -re -i tcp://127.0.0.1:1234?listen -f v4l2 /dev/video1

I know it is not a fix for the actual error, but it works nicely for what I need.

Thank you for your help.