Some question abount SharedImage on android platform

1,253 views
Skip to first unread message

Li Zehan

unread,
Dec 14, 2023, 10:58:19 AM12/14/23
to media-dev

Our project is a secondary development based on chromium（107 version）, and I am responsible for the media work.

Currently, I encountered a strange scenario, using webgl to obtain video content from the video element and then rendering it to canvas.  Demo link is: https://alphahans.github.io/index_ori_duration_100.html

On some Huawei mobile phones (Mate30pro\Mate40\Nova6...), you will find that the picture rendered by the video tag is stuck or rolled back. It seems that webgl copies the old texture and re-renders it on the video tag.

So I try to find the cause of the problem.


First of all, I found the unreasonable timing of wegbl calls. I replaced the following calls from
```
        // calls gl.drawArrays or gl.drawElements
        webglUtils.drawBufferInfo(gl, bufferInfo);
         
        // Tell WebGL we want to affect texture unit 0
        gl.activeTexture(gl.TEXTURE0);
        // Bind the texture to texture unit 0
        gl.bindTexture(gl.TEXTURE_2D, texture);
        // Tell the shader we bound the texture to texture unit 0
        gl.uniform1i( gl.getUniformLocation(meshProgramInfo.program, "uSampler"), 0);

       
        if( copyVideo ){
          gl.bindTexture(gl.TEXTURE_2D, texture);
          gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video);
        }else{
          // Fill the texture with a 1x1 blue pixel.
          gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, new Uint8Array([0, 0, 255, 255]));
        }
```
to
```
        if( copyVideo ){
          gl.bindTexture(gl.TEXTURE_2D, texture);
          gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video);
        }else{
          // Fill the texture with a 1x1 blue pixel.
          gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, new Uint8Array([0, 0, 255, 255]));
        }

              // Tell WebGL we want to affect texture unit 0
        gl.activeTexture(gl.TEXTURE0);
        // Bind the texture to texture unit 0
        gl.bindTexture(gl.TEXTURE_2D, texture);
        // Tell the shader we bound the texture to texture unit 0
        gl.uniform1i( gl.getUniformLocation(meshProgramInfo.program, "uSampler"), 0);

              // calls gl.drawArrays or gl.drawElements
        webglUtils.drawBufferInfo(gl, bufferInfo);
```
and found that the problem has been solved.

It is obvious here that webgl should render immediately after copying the texture, instead of waiting until the next cycle to render, otherwise the rendered picture will be old. Therefore, my guess as to why I faced the abnormal phenomenon above is that the video page draws old frames, causing <video> to roll back or freeze.

But I don't know why this problem can be solved here.

Previously, I was an audio and video developer(not in chromium),so I try to understand the code of media.


My Question:
1. Is the data decoded by MediaCodec converted into SharedImage a deep copy?
Why do I have this question?
Combing through the code, I found that webgl gets the latest videoframe through VideoFrameCompositor::GetCurrentFrameOnAnyThread. code like:
```
scoped_refptr<media::VideoFrame>
VideoFrameCompositor::GetCurrentFrameOnAnyThread() {
  base::AutoLock lock(current_frame_lock_);
  return first_video_frame_ ? first_video_frame_ : current_frame_;
}
```
I modified this to always return the first frame of the video after decoding. I expected that the canvas would always render the first frame of the video, but I found that the behavior was not what I expected.

2. Is there any documentation that I can learn more about VideoImageReaderImageBacking and VideoSurfaceTextureImageBacking?
Why do I have this question?
I found that if I don't adjust the code of webgl, I still keep the original way of calling it. Then I changed the default implementation of AndroidVideoImageBacking from VideoImageReaderImageBacking to VideoSurfaceTextureImageBacking and found that the problem of video tag playback lag or rollback was also solved.

3. Is there more documentation that can help me get familiar with Chromium's gpu-related mechanisms on Android?
Articles I have read:
https://docs.google.com/document/d/12qYPeN819JkdNGbPcKBA0rfPXSOIE3aIaQVrAZ4I1lM/edit?pli=1#heading=h.h8qur11p2eif
https://chromium.googlesource.com/chromium/src.git/+/master/docs/design/gpu_synchronization.md


Thank you for your help and I hope you work well.

Dale Curtis

unread,
Dec 14, 2023, 12:19:30 PM12/14/23
to Li Zehan, media-dev, Graphics-dev
Replies in line.

Android is a bit special in that frames are not discrete frames but instead mostly just pointers back to MediaCodec -- since you have to call MediaCodec.releaseOutputBuffer() to render frames. Frames can only be used in order on Android. You could modify MediaCodecVideoDecoder to always copy frames if you need such behavior and don't need to worry about the power costs of such a move.
 

2. Is there any documentation that I can learn more about VideoImageReaderImageBacking and VideoSurfaceTextureImageBacking?
Why do I have this question?
I found that if I don't adjust the code of webgl, I still keep the original way of calling it. Then I changed the default implementation of AndroidVideoImageBacking from VideoImageReaderImageBacking to VideoSurfaceTextureImageBacking and found that the problem of video tag playback lag or rollback was also solved.

+Graphics-dev in case they have additional pointers.
 



Thank you for your help and I hope you work well.

--
You received this message because you are subscribed to the Google Groups "media-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to media-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/media-dev/a82eaa15-8838-435e-9f82-16951e1e5dcbn%40chromium.org.
Reply all
Reply to author
Forward
0 new messages