Shared WebGL Resources

[Sebavan]

Hello,

Starting my journey about Machine Learning, I decided to give a go at TFJS cause I love WebGL :-)

My learning project is about a smart cam so nothing fancy about the Model part but I would like to optimize the project as much as I can and prevent all the extra copies/roundtrips "between CPU and GPU".

As all my effects are running with WebGL and TFJS is doing the same, I figure there should be some way to directly share resources between My app and TFJS (reusing the same context and textures for instance). Looking around in samples and so on, it does not look like there are any easy path for this scenario (here is one of the closest feature request I found about this: https://github.com/tensorflow/tfjs/issues/3457 and a pointer on context sharing: https://observablehq.com/@dsmilkov/using-custom-webgl-context)

I am wondering if this is something the community has looked into and if I am not simply missing the obvious ???

Thanks a lot in advance for your answers and sorry if I am not on the right channel to have this discussion (completely n00b in this community).

BR,

[Jason Mayes]

Congratulations on your TFJS adventures and big thumbs up for WebGL!

I shall have to delegate to one of the SWEs on the team to go into why that implementation detail is the way it is as I joined after this, but my gut feeling is that Canvas 2D was chosen for compatibility to maximise devices it can run on - though team - please correct me if I am wrong.

I am also interested in the answer here and if this is something that would be beneficial at some point in the future or not.

Thank you for bringing up the question - and look forward to seeing what you create - do share with us!

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/7a24b5e4-1cc0-4307-8241-4bbaae58450an%40tensorflow.org.

--

•  Jason Mayes   •  Senior Developer Advocate, TensorFlow.js   •  Google   •   Follow my discoveries or contribute your own

This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[Sebavan]

I think I am onto something with some help from the community which is of a great help !!!

Basically, this part helps setting the custom shared context (should double check version and so on):

const mygl = this.canvas.getContext('webgl2');

setWebGLContext(2, mygl as any);

await tf.setBackend('webgl');

This (Hackish one) ensures the restored state after I am done with my own work in the canvas (would be great as a restore helper):

const gpgpu = (tf.backend() as any).gpgpu;

const gl = this.engine._gl;

if (gpgpu.vertexAttrsAreBound) {

    this.engine.enableScissor();

    gl.useProgram(gpgpu.program);

    gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, gpgpu.indexBuffer);

    gpgpu_util.bindVertexProgramAttributeStreams(gl, false, gpgpu.program, gpgpu.vertexBuffer);

}

And this helps me sharing output internal tensor textures with my app (would also be great as a simpler helper):

const output = resize(inputTensor);

const backend = tf.backend() as unknown as tfgl.MathBackendWebGL;

const dataId = (output as any).dataId

const textureData = backend.texData.get(dataId);

const texture = textureData.texture;

The only missing part would be able to share inputs data by having a kind of fromTexture equivalent to fromPixels.

Any thoughts, concerns ?

[Sebavan]

Actually, we ll be continuing the discussion directly on Github if this gets any traction : https://github.com/tensorflow/tfjs/issues/1607 in order to dig into more specific details about any implementations that could happen.

Thanks a ton again for all your help.

BR,