Don't we need to release the outputbuffer of current frame, while previous frame texture is unavail?

[Jelari]

I have been looking around below file

https://webrtc.googlesource.com/src/+/master/sdk/android/src/java/org/webrtc/HardwareVideoDecoder.java

Also, i have been going through the  https://developer.android.com/reference/android/media/MediaCodec.html  to get a basic understanding of how codec is supposed to work. 

Two of the functions are really interesting. 

1) private void deliverTextureFrame(final int index, final MediaCodec.BufferInfo info, final int rotation, final Integer decodeTimeMs);

2) public void onTextureFrameAvailable(int oesTextureId, float[] transformMatrix, long timestampNs);

This  deliverTextureFrame() is invoked in the buzy while(running) loop.  Once we dequeueOutputBuffer() From the codec, we will try to invoke

releaseOutputBuffer() which essentially give output buffer back to the Codec, and also sends the buffer to surface. 

My question:

-----------------

1) According to the https://developer.android.com/reference/android/media/MediaCodec.html , it  is very clear that  "While you are not required to resubmit/release buffers immediately to the codec, holding onto input and/or output buffers may stall the codec, and this behavior is device dependent. Specifically, it is possible that a codec may hold off on generating output buffers until all outstanding buffers have been released/resubmitted. Therefore, try to hold onto to available buffers as little as possible."

Well, now let me  link the above statement with the source code. (see red colored)

private void deliverTextureFrame(final int index, final MediaCodec.BufferInfo info,

      final int rotation, final Integer decodeTimeMs) {

    // Load dimensions from shared memory under the dimension lock.

    final int width, height;

    synchronized (dimensionLock) {

      width = this.width;

      height = this.height;

    }

    synchronized (renderedTextureMetadataLock) {

      if (renderedTextureMetadata != null) {

        return; // We are still waiting for texture for the previous frame, drop this one.

      }

      renderedTextureMetadata = new DecodedTextureMetadata(

          width, height, rotation, info.presentationTimeUs, decodeTimeMs);

      codec.releaseOutputBuffer(index, true);

    }

The particular if condition  ie:

    if (renderedTextureMetadata != null) {

        return; // We are still waiting for texture for the previous frame, drop this one.

      }

says, that i have sent already one decoded frame to the surface for which i have not received onTextureFrameAvailable(), and hence i am dropping now. Please correct me if i am wrong here. Fair Enough.  

Literally, this if condition says that "Hey i dequeued the current frame, but wait, i have not received the texture of the previous frame, hence i am dropping now. But here we have not released the outputbuffer back to the codec.

It is clearly mentioned in doc "holding output buffers may stall the codec, and this behavior is device dependent". Though it is a device dependent, is it a bad idea to release the output buffer here ? ( please see blue color line below)

ie:

 if (renderedTextureMetadata != null) {

       codec.releaseOutputBuffer(index, false)

        return; // We are still waiting for texture for the previous frame, drop this one.

      }

2) For me in LollyPop,  my code often visits "We are still waiting for texture for the previous frame, drop this one", What would be the reason for delayed onTextureFrameAvailable() call?

[Sami Kalliomäki]

Good point, I've uploaded a CL to make this change: https://webrtc-review.googlesource.com/#/c/src/+/68520

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/d1114571-dba4-4c59-b5b1-2865975a1822%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[computerscience basis]

Thank You Very much Sami.