javacv merging image and mp3 creates a mp4 longer than the original mp3

[andrei paraschiv]

Hi,

I am trying to merge an audio mp3 with bit rate 192kbps with a jpg file.

The problem is the resulting mp4 is longer than the original jpeg. 

IplImage ipl = cvLoadImage(path2ImageFile);
int height = ipl.height();
int width = ipl.width();
if(height%2!=0) height = height+1;
if(width%2!=0) width = width+1;

OpenCVFrameConverter.ToIplImage grabberConverter = new OpenCVFrameConverter.ToIplImage();  
FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(path2OutputFile,width,height); 
FrameGrabber audioFileGrabber = new FFmpegFrameGrabber(path2AudioFile);
try 
{  
    audioFileGrabber.start();

    recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264 );//AV_CODEC_ID_VORBIS
    recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);//AV_CODEC_ID_MP3 //AV_CODEC_ID_AAC

    recorder.setFormat("mp4");  
    recorder.setAudioChannels(2);
    recorder.start();  

    Frame frame = null;
    //audioFileGrabber.getFrameNumber()
    while ((frame = audioFileGrabber.grabFrame())!=null) 
    {   recorder.record(grabberConverter.convert(ipl));  
        recorder.record(frame);
    }

    recorder.stop();  
    audioFileGrabber.stop();
 }  

Is there something I am doing wrong? Thank you

[Samuel Audet]

Since you have only one video frame, record it only once, then you'll have a file that is mostly audio only.

2020年5月11日(月) 7:52 andrei paraschiv <paraschiv...@gmail.com>:

[andrei paraschiv]

  audioFileGrabber = new FFmpegFrameGrabber(yad.getAudioFilePath());
        audioFileGrabber.start();
       
        recorder = new FFmpegFrameRecorder(path2OutputFile,width,height,audioFileGrabber.getAudioChannels());


        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264 );//AV_CODEC_ID_VORBIS
        //recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);//AV_CODEC_ID_MP3 //AV_CODEC_ID_AAC
        //recorder.setPixelFormat(PIX_FMT_YUV420P); //PIX_FMT_YUV420P      
        recorder.setFormat("mp4");  
       
        recorder.setFrameRate(35);
        recorder.setSampleRate(audioFileGrabber.getSampleRate());

       
        recorder.start();  
       


        Frame frame = null;

        int curImgNr = 0;
        long curTimestamp = 0;
        ipl = cvLoadImage(sortedImages.get(curImgNr).getImageFilePath());
        recorder.record(grabberConverter.convert(ipl));
        while ((frame = audioFileGrabber.grabFrame())!=null)
        {  
            curTimestamp = frame.timestamp;
           
           
            recorder.record(frame);
           
            if(curTimestamp >= sortedImages.get(curImgNr).getToSecondNr() * 1000000)
            {
                if(curImgNr < sortedImages.size() - 1)
                {
                    curImgNr++;
                    ipl = cvLoadImage(sortedImages.get(curImgNr).getImageFilePath());
                    recorder.record(grabberConverter.convert(ipl));
                }
                else break;
            }
             
        }
       
        recorder.stop();

If I do it this way.

All the images except for the very last one appear only for less than a second and then the last one appears for the rest of the video, but it looks really bad.

Or did you have something else in mind?

On Monday, May 11, 2020 at 2:10:06 AM UTC+3, Samuel Audet wrote:

Since you have only one video frame, record it only once, then you'll have a file that is mostly audio only.

2020年5月11日(月) 7:52 andrei paraschiv <paraschi...@gmail.com>:

[Samuel Audet]

You'll need to call setTimestamp() before the record() of each image.

[andrei paraschiv]

If I do that the video is just black. No images appear whatsoever.

       audioFileGrabber = new FFmpegFrameGrabber(yad.getAudioFilePath());
        audioFileGrabber.start();
       
        recorder = new FFmpegFrameRecorder(path2OutputFile,width,height,audioFileGrabber.getAudioChannels());


        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264 );//AV_CODEC_ID_VORBIS
        //recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);//AV_CODEC_ID_MP3 //AV_CODEC_ID_AAC
        //recorder.setPixelFormat(PIX_FMT_YUV420P); //PIX_FMT_YUV420P      
        recorder.setFormat("mp4");  
       
        recorder.setFrameRate(35);
        recorder.setSampleRate(audioFileGrabber.getSampleRate());
       
        recorder.start();  


        Frame frame = null;
        int curImgNr = 0;
        long curTimestamp = 0;
        ipl = cvLoadImage(sortedImages.get(curImgNr).getImageFilePath());

        recorder.setTimestamp(curTimestamp);

        recorder.record(grabberConverter.convert(ipl));
        while ((frame = audioFileGrabber.grabFrame())!=null)
        {  
            curTimestamp = frame.timestamp;


            recorder.record(frame);
           
            if(curTimestamp >= sortedImages.get(curImgNr).getToSecondNr() * 1000000)
            {

                recorder.record(grabberConverter.convert(ipl));

                if(curImgNr < sortedImages.size() - 1)
                {
                    curImgNr++;
                    ipl = cvLoadImage(sortedImages.get(curImgNr).getImageFilePath());

                    recorder.setTimestamp(curTimestamp);
                    recorder.record(grabberConverter.convert(ipl));
                }
                else break;
            }
        }
       
        recorder.stop();  
        audioFileGrabber.stop();
        audioFileGrabber.release();

[Samuel Audet]

The timestamps are in *microseconds*. Be sure you get the values right.

[andrei paraschiv]

well I obtain them like so

curTimestamp = frame.timestamp;

and thus in microseconds so I don't think that can be the issue

Something strange is going on.

In VLC in works. In the default windows media player it does not. It is all black. I am running windows 10 Pro. 

[Samuel Audet]

Ok, so it does work. It's just your player that doesn't support such low
frame rates. Anything less than 1 FPS is probably not a good idea...

[andrei paraschiv]

so can I fix it somehow?

[Samuel Audet]

Yes, try to increase the frame rate to 1 FPS or above.

2020年5月18日(月) 11:00 andrei paraschiv <paraschiv...@gmail.com>:

--

---
You received this message because you are subscribed to the Google Groups "javacv" group.
To unsubscribe from this group and stop receiving emails from it, send an email to javacv+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/javacv/26c0b60b-f42d-4625-8873-00e0869371d7%40googlegroups.com.

[andrei paraschiv]

But my frame rate is set at 35

audioFileGrabber = new FFmpegFrameGrabber(yad.getAudioFilePath());
        audioFileGrabber.start();
       
        recorder = new FFmpegFrameRecorder(path2OutputFile,width,height,audioFileGrabber.getAudioChannels());


        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264 );//AV_CODEC_ID_VORBIS
        //recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);//AV_CODEC_ID_MP3 //AV_CODEC_ID_AAC
        //recorder.setPixelFormat(PIX_FMT_YUV420P); //PIX_FMT_YUV420P      
        recorder.setFormat("mp4");  
       
        recorder.setFrameRate(35);
        recorder.setSampleRate(audioFileGrabber.getSampleRate());
       
        recorder.start();  

On Monday, May 18, 2020 at 5:28:13 AM UTC+3, Samuel Audet wrote:

Yes, try to increase the frame rate to 1 FPS or above.

2020年5月18日(月) 11:00 andrei paraschiv <paraschi...@gmail.com>:

so can I fix it somehow?

On Monday, May 18, 2020 at 4:39:21 AM UTC+3, Samuel Audet wrote:

Ok, so it does work. It's just your player that doesn't support such low
frame rates. Anything less than 1 FPS is probably not a good idea...

On 5/18/20 10:37 AM, andrei paraschiv wrote:
> Something strange is going on.
> In VLC in works. In the default windows media player it does not. It is
> all black. I am running windows 10 Pro.

--

---
You received this message because you are subscribed to the Google Groups "javacv" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jav...@googlegroups.com.

[Samuel Audet]

That only sets the maximum frame rate. Once we start calling
setTimestamp() manually, we get a variable frame rate.

[andrei paraschiv]

So I modified my code this way

audioFileGrabber = new FFmpegFrameGrabber(yad.getAudioFilePath());
        audioFileGrabber.start();
       
        recorder = new FFmpegFrameRecorder(path2OutputFile,width,height,audioFileGrabber.getAudioChannels());


        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264 );//AV_CODEC_ID_VORBIS
        //recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);//AV_CODEC_ID_MP3 //AV_CODEC_ID_AAC
        //recorder.setPixelFormat(PIX_FMT_YUV420P); //PIX_FMT_YUV420P      
        recorder.setFormat("mp4");

 
        recorder.setVideoQuality(0);

        recorder.setFrameRate(35);
        recorder.setSampleRate(audioFileGrabber.getSampleRate());
       
        recorder.start();  


        Frame frame = null;
        int curImgNr = 0;
        long curTimestamp = 0;
        ipl = cvLoadImage(sortedImages.get(curImgNr).getImageFilePath());
        recorder.setTimestamp(curTimestamp);
        recorder.record(grabberConverter.convert(ipl));
        while ((frame = audioFileGrabber.grabFrame())!=null)
        {  
            curTimestamp = frame.timestamp;


            recorder.record(frame);
           
            if(curTimestamp >= sortedImages.get(curImgNr).getToSecondNr() * 1000000)
            {

                recorder.setTimestamp(curTimestamp);

                recorder.record(grabberConverter.convert(ipl));
                if(curImgNr < sortedImages.size() - 1)
                {
                    curImgNr++;
                    ipl = cvLoadImage(sortedImages.get(curImgNr).getImageFilePath());
                    recorder.setTimestamp(curTimestamp);
                    recorder.record(grabberConverter.convert(ipl));
                }
                else break;
            }
        }
       
        recorder.stop();

 
        recorder.release();
        audioFileGrabber.stop();
        audioFileGrabber.release();

I call setTimestamp twice for each image. It's still black in the standard win 10 movie player. Should I call it some more? Maybe 33 more times per second?

Should I check if a 60/33 seconds have passed since the last call to setTimestamp and call setTimestamp agian?

Is that what you are saying?

Or am I misunderstanding?

Thank you very much for the help so far.

[Samuel Audet]

Yes, something like that. Make sure it works at standard 30 FPS with setFrameRate(30) without calling setTimestamp(). If that doesn't work, the problem is somewhere else.

2020年5月19日(火) 23:35 andrei paraschiv <paraschiv...@gmail.com>:

--

---
You received this message because you are subscribed to the Google Groups "javacv" group.

To unsubscribe from this group and stop receiving emails from it, send an email to javacv+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/javacv/1bae5468-93f0-4cd0-87e2-435f530fd145%40googlegroups.com.

[andrei paraschiv]

just calling setFrameRate(30) doesn't do anything 

I tried doing this:

 audioFileGrabber = new FFmpegFrameGrabber(yad.getAudioFilePath());
        audioFileGrabber.start();
       
        recorder = new FFmpegFrameRecorder(path2OutputFile,width,height,audioFileGrabber.getAudioChannels());


        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264 );//AV_CODEC_ID_VORBIS
        //recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);//AV_CODEC_ID_MP3 //AV_CODEC_ID_AAC
        //recorder.setPixelFormat(PIX_FMT_YUV420P); //PIX_FMT_YUV420P      
        recorder.setFormat("mp4");  
        recorder.setVideoQuality(0);

        recorder.setFrameRate(maxFrameRate);

        recorder.setSampleRate(audioFileGrabber.getSampleRate());
       
        recorder.start();  


        Frame frame = null;

        int curImgNr = 0,picFramesInsertedSinceLastClean=0;
        long curTimestamp = 0;
        long lastPicInsertTimestamp = 0;

        ipl = cvLoadImage(sortedImages.get(curImgNr).getImageFilePath());
        recorder.setTimestamp(curTimestamp);
        recorder.record(grabberConverter.convert(ipl));

        picFramesInsertedSinceLastClean++;

        while ((frame = audioFileGrabber.grabFrame())!=null)
        {  
            curTimestamp = frame.timestamp;
 
            recorder.record(frame);
           
            if(curTimestamp >= sortedImages.get(curImgNr).getToSecondNr() * 1000000)
            {
                recorder.setTimestamp(curTimestamp);
                recorder.record(grabberConverter.convert(ipl));

                lastPicInsertTimestamp = curTimestamp;
                picFramesInsertedSinceLastClean++;

               
                if(curImgNr < sortedImages.size() - 1)
                {
                    curImgNr++;
                    ipl = cvLoadImage(sortedImages.get(curImgNr).getImageFilePath());
                    recorder.setTimestamp(curTimestamp);
                    recorder.record(grabberConverter.convert(ipl));

                    lastPicInsertTimestamp = curTimestamp;
                    picFramesInsertedSinceLastClean++;
                }
                else break;
            }
           
            if(curTimestamp >= lastPicInsertTimestamp + (1000000 / maxFrameRate))//1 sec/maxnrframes passed
            {
                if(picFramesInsertedSinceLastClean < maxFrameRate)
                {
                    recorder.setTimestamp(curTimestamp);
                    recorder.record(grabberConverter.convert(ipl));
                    lastPicInsertTimestamp = curTimestamp;
                    picFramesInsertedSinceLastClean++;
                }
                else
                {
                    picFramesInsertedSinceLastClean = 0;
                }
            }
        }
       
        if(curTimestamp < sortedImages.get(curImgNr).getToSecondNr() * 1000000 && picFramesInsertedSinceLastClean < maxFrameRate)
        {
            recorder.setTimestamp(curTimestamp);
            recorder.record(grabberConverter.convert(ipl));
        }
       
        recorder.stop();  
        recorder.release();
        audioFileGrabber.stop();
        audioFileGrabber.release();

When it reached recorder.stop() it throws this error "Exception: av_interleaved_write_frame() error -22 while writing interleaved video packet."

Am I doing something wrong?

On Wednesday, May 20, 2020 at 2:16:07 AM UTC+3, Samuel Audet wrote:

Yes, something like that. Make sure it works at standard 30 FPS with setFrameRate(30) without calling setTimestamp(). If that doesn't work, the problem is somewhere else.

2020年5月19日(火) 23:35 andrei paraschiv <paraschi...@gmail.com>:

To unsubscribe from this group and stop receiving emails from it, send an email to jav...@googlegroups.com.

[Samuel Audet]

Check the error message in the log. You'll probably need to call record(image) more often.

2020年5月21日(木) 8:50 andrei paraschiv <paraschiv...@gmail.com>:

To unsubscribe from this group and stop receiving emails from it, send an email to javacv+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/javacv/12274510-ea06-4ea5-aafe-8cd548a692b1%40googlegroups.com.