Loading WebGL textures from downloaded images

[Phil Endecott]

Dear List,

I've been using Emscripten to port an OpenGL project and I am
impressed by how well it works, so far.

My original code decompresses JPEG images and loads them into
textures on background threads. For Emscripten I've disabled that,
and it does impact performance.

I am wondering whether the best solution to this would be to
download, decode and load the images in Javascript. Specifically,
MDN has an example here:

https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/Tutorial/Using_textures_in_WebGL

Basically it creates an Image object, sets its src attribute
to the URL to download, and sets an onload handler that calls
gl.texImage2D to load it into a texture.

Potentially, this might use more efficient JPEG decoding and
might do that decoding on a different thread.

Has anyone tried anything like this? Does anyone know whether
browsers do, in practice, use secondary threads and/or faster
decoders than libjpeg-turbo, for decoding JPEGs?


Thanks,

Phil.

[Shachar Langbeheim]

Yes, that's exactly what we do in https://app.boosted.lightricks.com/ . We benchmarked the various ways to move the texture information from the JS FE to the C++ OpenGL implementation, and the fastest method was to create an HTMLImage/VideoElement and load the texture content using texImage2D, and then just to pass the texture handle to the WASM code. We did need to copy the way that Emscripten gives each texture a unique handle , in order to access the textures in C++.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/1613328173462%40dmwebmail.dmwebmail.chezphil.org.

[Shachar Langbeheim]

BTW, it's not only more efficient, but it also ensures that the browser will handle things like EXIF rotation for you.

[Jukka Jylänki]

Reusing browser codecs to load images is a common tactic in web
builds. It allows shrinking generated code by quite a bit by not
having to ship image decoders in wasm, and browsers do parallelize
image decoding since web pages are generally filled with images.

See here for a small example: https://github.com/juj/webgl_render_test
, in particular this function that does the uploading:
https://github.com/juj/webgl_render_test/blob/bb517e34dfaa13404feaa8a5ba8bf42c32581c44/src/library_js.js#L32

Online version available here: http://clb.confined.space/greetings/

With respect to performance, it is tough to say whether Wasm
threading+custom image decoders would be more performant than JS Image
based decoders - that is something I haven't benchmarked in detail. It
probably depends on the nuances for the kinds of features that one
needs for the decoding. (CPU access, processing, access to metadata
info, etc.)

> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CA%2B_KjGbdD0tn-N2vxSZfsOs3%3DLMc_m%2BibPCnb82%2BN3rsnysukQ%40mail.gmail.com.

[Shachar Langbeheim]

>  it is tough to say whether Wasm threading+custom image decoders would be more performant than JS Image based decoders

AFAIK This is going to vary significantly between the files and hardware - it very much depends on whether your setup has HW codecs for the specific video format you us. This might be a lesser consideration for images, but when decoding video and trying to reach 60 FPS it's a pretty big deal.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CA%2B6sJ-2%3DODyF4Wsq8%2BQL50p2oJwaH%3DcwCVc%3DzRGoSqzM3%2Bfibw%40mail.gmail.com.

[Phil Endecott]

Thanks everyone for your useful replies.

Jukka Jylänki wrote:
> See here for a small example: https://github.com/juj/webgl_render_test
> , in particular this function that does the uploading:
> https://github.com/juj/webgl_render_test/blob/bb517e34dfaa13404feaa8a5ba8bf42c32581c44/src/library_js.js#L32

Right, that's close to what I need. I have a couple of questions:

1. Can anyone suggest how I can add a callback from the Javascript
to the C++ when the texture is loaded? I can't see a simple JS
call to which I can pass a C function pointer. (Or something
using a std::function?)

2. Any ideas about how to cancel an in-progress load? Maybe
deleting the Image object will abandon any in-progress download?

Pseudo-code:

// fetch_texture.js

fetch_texture: function(url, texturenum, callback)
{
url = UTF8ToString*url);
const texture = GL.textures[texturenum];
const image = new Image();
image.onload = function() {
GLctx.bindTexture(GLctx.TEXTURE_2D, texture);
GLctx.texImage2D(GLctx.TEXTURE_2D, 0, GLctx.RGBA, GLctx.RGBA, GLctx.UNSIGNED_BYTE, image);
callback(); // <---- ???
};
image.src = url;
return handle_to_image; // <--- ???
}

cancel_fetch_texture: function(handle)
{
delete image_from_handle; // ????
}

// fetch_texture.hh

extern "C" {
handle_t fetch_texture(const char* url, int texturenum, void(*callback)());
void cancel_fetch_texture(handle_t handle);
};


Many thanks, Phil.

[Jukka Jylänki]

Search for a string snippet "{{{ makeDynCall(" in Emscripten's src/
subdirectory for examples how to call a C/C++ function pointer from a
JS library. For example here:
https://github.com/emscripten-core/emscripten/blob/a38a72af8a75996fce5e27fcfb45f17ac38a0ce6/src/library_html5.js#L2723

In JavaScript "delete image_from_handle;" for a variable is a no-op.
The only syntax is "delete dict[key];" which will remove key from JS
object dict. You can try first setting an image.src to an empty string
''; and then dropping all references to it, a smart browser should be
able to drop the transfer then, if it has not yet finished. For
explicit control, you could do a Fetch to an object URL, though that
is a bit more involved code flow.

ma 15. helmik. 2021 klo 17.54 'Phil Endecott' via emscripten-discuss
(emscripte...@googlegroups.com) kirjoitti:

> --
> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/1613404449510%40dmwebmail.dmwebmail.chezphil.org.

[Phil Endecott]

Jukka Jylänki wrote:
> In JavaScript "delete image_from_handle;" for a variable is a no-op.
> The only syntax is "delete dict[key];" which will remove key from JS
> object dict. You can try first setting an image.src to an empty string
> ''; and then dropping all references to it, a smart browser should be
> able to drop the transfer then, if it has not yet finished.

A quick check in Safari suggests that it is not cancelling the
download when the src is set to ''. On the other hand, Firefox
probably is; I see "NS_BINDING_ABORTED" in the networking inspector.

Anyway, this is basically working. Thanks everyone for your help.


Phil.