canvaskit-wasm: understanding makeImageSnapshot() memory usage in WebGL

[Alan Song]

We have a tile renderer that works like this: 

Suppose there is 2048*2048 px content, divided into 8*8 tiles each of size 256*256.

We initially create a surface with MakeRenderTarget of 2048*2048 to render everything, and loop from top-left to right-bottom to call makeImageSnapshot() on each 256*256 tile, save the snapshot image to cache. Finally, we loop through them, call drawImage and flush to the on-screen canvas.

For updates to the scene, say a 1024*1024 (4*4 tiles) area, we delete the dirty tiles (skimage.delete()), and then create another render target of 1024*1024, render and extract the 4*4 tiles, and flush to screen.

We observed that with this approach, memory usage is much higher than expected. Naively, each tile would take 256*256*4(one byte for each of rgba) = 0.25MB memory. 64 tiles would take 16MB. However, the GPU Process in task manager shows much larger memory used. (If only there’s proper memory profiling tool for webgl...)

Reproduction link:  https://jsfiddle.net/wxyo8ugz/

1. Open the page, take the baseline memory in Chrome Task Manager

Memory Footprint: 145MB, GPU Memory: 52.6MB

2. Click “render first frame” which renders 2048*2048 content and extracts tiles

 First a spike of Memory Footprint: 264MB, GPU Memory: 111MB

Eventually it settles down to Memory Footprint: 204MB, GPU Memory: 69.5MB

If Memory Footprint represents total memory, and GPU Memory means textures uploaded to GPU, then the delta is
204-145 = 59MB extra total memory (instead of 16MB)
69.5-52.6 = 16.9MB extra uploaded to GPU (coincides with calculated 16MB size, but I doubt it’s accurate, see next step)

3. Click “render second frame” which renders 1024*1024 content and extracts tiles

Initial spike is Memory Footprint: 264MB, but somehow GPU memory reduces to 53.5MB.  

Afterwards Memory Footprint goes to 193MB and GPU Memory 53.0MB.

This is kind of surprising because GPU Memory decreased, so I added extra code to rerender the frame on mouse move. 

After moving the mouse for a while. Memory Footprint is 245MB and GPU memory is 101MB, so the delta is

245-145 = 100MB
101-52.6 = 48.4MB

In the actual implementation we use drawImageRectCubic for scaling the tiles so there should be no mipmap involved (and its *1.33 for memory usage).

So:

1. Is it wrong to assume skImage.delete() would delete the corresponding raster data? Is it instead the case that the whole surface’s raster data is snapshotted, and makeImageSnapshot with bounds just creates a reference (that gets removed on delete())? And the whole surface data won't be deleted unless all snapshots of it are deleted?
2. Can we assume the same width*height*rgba memory size for MakeRenderTarget? Can the memory spike be avoided?

Thank you!