Creating video from still images

[Darin Dimitrov]

I am trying to create video using MediaStreamTrackGenerator and a couple of still images:

async function createVideo() {

    const trackGenerator = new MediaStreamTrackGenerator({ kind: 'video' });
    const stream = new MediaStream();
    stream.addTrack(trackGenerator);

    const recordedChunks = [];
    const mediaRecorder = new MediaRecorder(stream, { mimeType: 'video/webm;codecs=vp9' });
    mediaRecorder.ondataavailable = event => {
        if (event.data.size > 0) {
            recordedChunks.push(event.data);
        } else {
            console.error('Empty chunk');
        }
    };
    mediaRecorder.start();

    for (let i = 0; i < 10; i++) {
        const writer = trackGenerator.writable.getWriter();
        await writer.ready;

        const height = 640;
        const width = 480;
        const duration = 1000 * 1000;
        const timestamp = i * duration;
        const canvas = new OffscreenCanvas(width, height);
        const ctx = canvas.getContext('2d', { alpha: false });
        ctx.fillStyle = `rgba(50, 100, 150, 255)`;
        ctx.fillRect(0, 0, width, height);
        const frame = new VideoFrame(canvas.transferToImageBitmap(), {
          duration,
          timestamp,
          alpha: 'keep' ,
          codedWidth: width,
          codedHeight: height
        });

        await writer.write(frame);
        frame.close();
        await writer.ready;
        writer.releaseLock();
    }

    mediaRecorder.stop();

    const blob = new Blob(recordedChunks, {
        type: 'video/webm'
    });

    const url = URL.createObjectURL(blob);
    const a = document.createElement('a');
    document.body.appendChild(a);
    a.style = 'display: none';
    a.href = url;
    a.download = 'test.webm';
    a.click();
    window.URL.revokeObjectURL(url);
}

The problem is that if I write the images quickly in a loop (as shown in my code), the ondataavailable event is never raised and I get an empty blob. If on the other hand I delay the writing of each frame with 1second, then I get correct results.

So my question is if it is possible to create a video by batch adding the frames with their respective timestamp and duration?

[guest271314]

Why do you call 

   const writer = trackGenerator.writable.getWriter();

multiple times in a loop?

An approach using ImageBitmap https://plnkr.co/edit/4Tb91b?preview and one using ImageCapture API https://github.com/guest271314/MediaFragmentRecorder/tree/imagecapture-audiocontext-readablestream-writablestream.

[Darin Dimitrov]

I have tried also with getting the writer outside the loop and reusing it which didn't seem to change anything. 

As far as the examples you have shown are concerned, they seem to rely on playing the entire video through a <video> element. This means that in order to get the resulting video, I need to wait for the entire video to play.

I am looking for a way to assemble the resulting video as fast as possible from the given frames and timestamps.

[guest271314]

> I have tried also with getting the writer outside the loop and reusing it which didn't seem to change anything. 

You are creating multiple instances of WriteableStreamDefaultWriter. Only 1 is necessary.

> As far as the examples you have shown are concerned, they seem to rely on playing the entire video through a <video> element. 

The plnkr writes images already stored in a file that are fetched.

Are you trying to 

a) record a live MediaStreamTrack; or 

b) assemble images you already have to a media container or file?

> This means that in order to get the resulting video, I need to wait for the entire video to play.

You are currently doing that anyway with MediaStreamTrackGenerator, which is a live MediaStreamTrack.

--

---
You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/Fr6M5ha2sC4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/fdc14bc1-38da-4d2b-9ee7-5dbed2ffde02n%40googlegroups.com.

[guest271314]

> I am looking for a way to assemble the resulting video as fast as possible from the given frames and timestamps.

AFAIK no Web API provides a means to do that. E.g., see https://github.com/w3c/mediacapture-record/issues/166.

If you already have the images you can use webm-writer.js https://github.com/thenickdude/webm-writer-js. In this plnkr https://plnkr.co/edit/Inb676?preview I capture a video to WebM. If you have the images, you can call addFrame(). Note that in the code when resolution is variable, frame durations changes to for "seamless" transition between variable resolution and duration framed rendering on a screen.

You should also be able to adjust the code in https://github.com/davedoesdev/webm-muxer.js ("This project provides a way for your application to get its audio and video tracks into a WebM container.") to achieve the requirement.

[Darin Dimitrov]

Thanks for the pointers. Yes, I already have the images and would like to assemble them in a container. I have looked at webm-writer.js and it seems that it provides exactly the functionality I need. There is just one small caveat: it uses VP8 encoding (due to WebP) but I can try to modify it and use WebCodecs VideoEncoder to encode the image with VP9.