WebCodecs Videoencoder Pixelformat

[Marten Richter]

Hi,
I am doint my first experiments with the WebCodecs API, really love it.
Of course I have ran into the first issue...
I tested encoding and set the prefer hardware option for avc. (I know that this is not recommended...)
However, I got during encode 'OperationError Encoding error.' error. (It is an old nividia card in the system).
I have the suspicion, that the reason for the failure is the pixel format of the Frame , which is IV420, and the hardware decoder outputs NV12.
Is it possible, that it fails since the Pixelformat is not supported by the encoder?
How can I query the supported Pixel formats, in order to convert it if necessary? As far as I understood, it is not possible to use ' isConfigSupported' to figure this out... or should the browser handle this ?

Here are some details:

  this.videocodec.configure({

        codec: 'avc1.42402A' ,

        avc: { format: 'annexb' },

        framerate: 30,

        displayWidth: chunk.displayWidth,

        displayHeight: chunk.displayHeight,

        width: chunk.codedWidth,

        height: chunk.codedHeight,

        hardwareAcceleration: 'prefer-hardware',

        bitrate: 1000_000,

        scalabilityMode: 'L1T3', // does not have an impact

        latencyMode: 'realtime'

      })

and this is a typical video frame input: format: 'I420', timestamp: 607986, duration: null, codedWidth: 640, codedHeight: 360,

Thanks,

Marten

[Dan Sanders]

Is it possible, that it fails since the Pixelformat is not supported by the encoder?

 

I don't think this is the case, I420 is one of the most common capture formats so we routinely encode I420 frames.

If you're making these frames from ArrayBuffers (rather than something that would internally be backed by a texture), and NV12 does work, it's possible that there is a conversion we have missed which would be a bug in Chrome.

How can I query the supported Pixel formats, in order to convert it if necessary? As far as I understood, it is not possible to use ' isConfigSupported' to figure this out... or should the browser handle this ?

There is currently no way to query supported pixel formats. We take the view that all renderable formats should work, but some formats may be more efficient than others. This implies that if you can construct a VideoFrame, it should work for encoding. This isn't quite settled in the specification, though.

My recommendation is to look at the media log (DevTools > ... > More tools > Media) to see if there is a more descriptive message. Also check chrome://gpu in case it's something logged there. Failing either of those we would need a reproduction case to investigate.

- Dan

[Marten Richter]

Thanks for the reply. 
Now, I tried it again and it worked, without any change, but then at some time it failed again. So I agree it is probably not the  Pixelformat.
Anyway, I looked at the media tools log and it says:

{"causes":[],"code":7,"data":{},"group":"EncoderStatusCodes","message":"Failed to initialize video encode accelerator.","stack":[{"file":"media/video/video_encode_accelerator_adapter.cc","line":247}]} (but it is on current edge, will check later on chrome)

The frames are coming from a WebCam through a MedieTrack on a worker.
Since it fails only after a couple of refreshes, I could imagine some exhausted resources

Anyway I will prepare on the weekend, a working example on chrome..., 

Thanks,

Marten

[Marten Richter]

Ok, now everything for testing is set up.
It is on google chrome 99.0.4844.84

The message is the same as on edge:

{"causes":[],"code":7,"data":{},"group":"EncoderStatusCodes","message":"Failed to initialize video encode accelerator.","stack":[{"file":"media/video/video_encode_accelerator_adapter.cc","line":247}]}

It is failing here

if (!accelerator_->Initialize(vea_config, this)) {
InitCompleted(
EncoderStatus(EncoderStatus::Codes::kEncoderInitializationError, <=== this line is 247
"Failed to initialize video encode accelerator."));
return;

It is only failing after a couple of refreshes, then chrome is in a state so, that I can only restart it. Biggest problem I have created a simple example, but it can not bring chrome into this failed state, it only fails if my main app has done this (it is quite heavy reagarding many svg objects etc..., so GPU mem exhaustion may be the reason..., the other point is, it uses additionally a video decoder, so this might cause some interference). Also after I close my app tab the test prevails and the simple test also still fails, but it can not bring the browser into this state. Here ist the test code: (based on one of the webcodec tests);

Marten

Testcode:

<head>
<title>WebCodecs webcam stream in Worker</title>
</head>
Nothing, look into the console.log

<script>
  let frameCount = 0
function start() {

  if (typeof MediaStreamTrackProcessor === 'undefined' ||
    typeof MediaStreamTrackGenerator === 'undefined') {
    console.log(
      'Your browser does not support the experimental MediaStreamTrack API. ' +
      'Please launch with the --enable-blink-features=WebCodecs,MediaStreamInsertableStreams flag');
    return;
  }

  var constraints = {
    audio: false,
    video: {
      width: 1280,
      height: 720
    }
  };
  console.log('dom contentloaded')

  // Get a MediaStream from the webcam.
  navigator.mediaDevices.getUserMedia(constraints).then(function(mediaStream) {
    console.log('getusermedia')


    // Create a MediaStreamTrackProcessor, which exposes frames from the track
    // as a ReadableStream of VideoFrames.
    var track = mediaStream.getVideoTracks()[0];
    var processor = new MediaStreamTrackProcessor(track);


    var frameStream = processor.readable;

    const frameReader = frameStream.getReader();


    const output = (frame) => {
      console.log('got frame', frame);
    }

    const videocodec = new VideoEncoder({
      output: output,
      error(error) {
        console.log('encoder error', error.name, error.message)
      }
    })
    const cur = {} // current frame data
    console.log('before frame reader')

    frameReader.read().then(function processFrame({
      done,
      value
    }) {
      console.log('process frame', value)
      if (done) {
        console.log("Stream is done");
        return;
      }

      var frame = value;

      if (
        frame.displayHeight !== cur.displayHeight ||
        frame.displayWidth !== cur.displayWidth ||
        frame.codedHeight !== cur.height ||
        frame.codedWidth !== cur.width
      ) {
        const curcodec = 'avc1.42402A'
        videocodec.configure({
          codec: curcodec /* 'avc1.420034' */ , // aka h264, maybe add profile
          avc: {
            format: 'annexb'
          },
          framerate: 25,
          displayWidth: frame.displayWidth,
          displayHeight: frame.displayHeight,
          width: frame.codedWidth,
          height: frame.codedHeight,
          hardwareAcceleration: 'prefer-hardware',
          bitrate: 300000,
          scalabilityMode: 'L1T3',
          latencyMode: 'realtime'
        })
        cur.displayHeight = frame.displayHeight
        cur.displayWidth = frame.displayWidth
        cur.height = frame.codedHeight
        cur.width = frame.codedWidth
        console.log('current config', cur)

      }

      // NOTE: all paths below must call frame.close(). Otherwise, the GC won't
      // be fast enoug to recollect VideoFrames, and decoding can stall.



      videocodec.encode(frame, {
        keyFrame: (frameCount % 60 == 0)
      })

      // Processing on 'frame' goes here!
      // E.g. this is where encoding via a VideoEncoder could be set up, or
      // rendering to an OffscreenCanvas.

      // For now, simply confirm we are receiving frames.
      if (++frameCount % 20 == 0) {
        console.log("Read 20 frames");
      }

      frame.close();
      frameReader.read().then(processFrame);

    }).catch(function(err) {
      console.log(err.name + ": " + err.message)
    });
    console.log('after frame reader')
  }, false).catch((error) => {
    console.log('error caught', error)
  });
  console.log('after getusermedia')
}
start()
</script>

[Eugene Zemtsov]

Hi Marten,

We've observed that on Windows, NVIDIA drivers often won't allow more than 3 accelerated video encoders per process (in our case it's chrome GPU). 

Do you by any chance keep more than one tab open that are using accelerated video encoding? 

[Marten Richter]

Hi Eugene,

no I do not have more then one tab with video encoders open (in one chrome instance).
While testing I had always just one tab for testing and when I was refreshing several times at some point it did not work anymore.
I have seen the encoder limit in the chromium source, but I am not 100% convinced that it is really just the encoder that exhausted.

Since  it only happend on the big app not on the small testing code. 
Anyhow I believe, that some resource is not freed, when refreshing (may be the resources are not closed properly).
But when closing the browser, the problem goes away....

Thanks...

Marten

[Eugene Zemtsov]

Do you always call close() on encoder objects?