Pixel by Pixel 2D rendering in WebGL

[Chad Stearns]

Hi there,

So Ive got a pretty weird use case, where I want to render onto a 2D view, and I need to specify individual pixel color values. I am pretty familiar with how canvas elements work. Im trying to study up on web-gl, and I understand that its more performant by accessing the clients GPU. Ive got some buggy code that doesnt work right now that renders a flat surface directly in front of the view. Two questions..

0 Since I just want to specify pixel color data, are there actually any performance gains for my use case by using web-gl?

1 Is there a way to do this? How? What I really want in an api where I say something like 'change the color at pixel (x, y) to color B'. Is there any way to use web-gl to do this in a way that has performance advantages from just doing that to a canvas element directly?

Thank you so much,

-Chad

[Kevin Reid]

On Oct 14, 2016, at 17:11, Chad Stearns <chadst...@gmail.com> wrote:
> So Ive got a pretty weird use case, where I want to render onto a 2D view, and I need to specify individual pixel color values. I am pretty familiar with how canvas elements work. Im trying to study up on web-gl, and I understand that its more performant by accessing the clients GPU. Ive got some buggy code that doesnt work right now that renders a flat surface directly in front of the view. Two questions..
>
> 0 Since I just want to specify pixel color data, are there actually any performance gains for my use case by using web-gl?

If you can rewrite your "specifying pixel colors" as a GLSL shader, so that a single shader computes many pixels' colors (on the GPU), then you can get a very large benefit from WebGL.

If your code has to run on the CPU, but you can arrange to write the color values of many pixels into an array (that makes up an image) to be drawn all at once as an image/texture, then this will have decent performance and you can use _either_ 2D-context (ImageData) or WebGL (textures) to do it.

If you really, really have to change lots of non-adjacent pixels at a time, you should use 2D-context because it will be more efficient at that than WebGL.

> 1 Is there a way to do this? How? What I really want in an api where I say something like 'change the color at pixel (x, y) to color B'. Is there any way to use web-gl to do this in a way that has performance advantages from just doing that to a canvas element directly?

In general, "set this pixel to this color" is the _least_ efficient case for _any_ graphics API. You are much better off figuring out how to draw whatever you're drawing in bigger batches.


If you gave more information about exactly what you are drawing and how it is computed, I could be more specific about how you might efficiently draw it.

[Chad Stearns]

Hey Kevin,

Im interesting in making an in-browser image editor. I worked on a project doing this a few years back. I just used a canas element without web-gl (ctpaint.us/app). Other people have done similar things, like this one by a guy named Alex (https://prominentdetail.github.io/Pixel.Tools/).

Im thinking about doing another in-browser image editor project. I want to make it as efficient as possible. So I was considering learning web-gl. So, the user very plausibly will be modifying non-adjacent pixels. Or maybe drawing and modifying just one pixel at a time, such as with a pencil tool, or a line tool.

Im coming to believe that web-gl really cant help. Its advantage is that it does graphics computing very efficiently, while in my use case, whats to be rendered is basically already computed. Namely I have some model of what the canvas data is, and I just need to put that into the canvas element, rather than needing to do any heavy calculating of how that model translated into a rendering. Am I correct in my assessment?

Thank you so much,

-Chad

[Andre Weissflog]

I do this in my 8-bit emulator (http://floooh.github.io/virtualkc/), this is using WebGL (through emscripten's GL wrapper API, so I can't point you to JS code, only C++), but it works like this:

I'm creating 2 double-buffered empty textures in RGBA8 format (1 pixel == 4 bytes), and a linear uint32 array where the CPU writes to (the backing buffer where your unique pixel-writes would go to). Every frame the CPU-backing-buffer is copied to one of the RGBA8 textures via texSubImage2D (alternating the texture each frame to prevent a sync-stall, I'm not 100% sure if this is necessary on WebGL, but my code also needs to work on native GL platforms).

Then I'm rendering a fullscreen quad through WebGL where the texture that's just been filled with texSubImage2D() is rendered to screen through a fragment shader which in the easiest case just samples the texture, and in more complex cases can apply image effects (that's the main advantage of using a WebGL canvas in this scenario).

As far as I can see from my emu, this is about as performant on WebGL as on native platforms.

Cheers,

-Floh.

[Kevin Reid]

On Oct 15, 2016, at 16:11, Chad Stearns <chadst...@gmail.com> wrote:
> Im thinking about doing another in-browser image editor project. I want to make it as efficient as possible. So I was considering learning web-gl. So, the user very plausibly will be modifying non-adjacent pixels. Or maybe drawing and modifying just one pixel at a time, such as with a pencil tool, or a line tool.
>
> Im coming to believe that web-gl really cant help.

Actually, this sounds like a perfectly reasonable thing to do with WebGL. I said that changing individual pixels is the least efficient case, yes, but in this case;

(a) what you're changing that sort of way is limited to the user's mouse/pen/whatever input (not very much per frame), and

(b) drawing a line is _not_ setting individual pixels, it is drawing a whole line.

If you were to use WebGL for this purpose, you would keep the user's image in a texture (with an autosaved backup in regular memory, because context loss is a thing that happens) and each time there's some input, like the line segment which is the latest movement of the pen tool, you would render _to_ a texture the new “frame” which is just the original texture with the new content painted on top of it. Then you use the second texture as the input for the next update writing to the first texture, and so on.

For simple things, this is more work than using 2D-context but it will work way better for anything fancy like a soft-edged brush, because you can have a fragment shader compute the effect you want per-pixel rather than trying to build it out of vector shapes and composite operations.

[Olli Etuaho]

There's a library I wrote for this kind of an use case. It's kind of a hybrid vector/bitmap brush engine, it supports resizing the canvas and unlimited undo for example. I worked on it for a few years, haven't added new features lately but it's still pretty solid:

https://github.com/Oletus/glbrush.js

Example app: http://oletus.github.io/glbrush.js/example.html

And a more fully fledged app using the library: http://www.coopaint.com/

[Chad Stearns]

So I understand that a lot of drawing applications are using brushes that do complicated brush patterns that require a lot of computation, and I can see how web-gl would be useful for that. The paint software I am interested in making would be a lot more like MSPaint and would be suited for making pixel art. Complicated brushes arent a feature of MSPaint, and wouldnt be a feature of the paint software I would make. Are you saying web-gl is only advantageous for rendering brush strokes? What if the 'brush' really is just one pixel changing to a different color? Or something comparably simple, like filling in a square area with a specific color. You insisted that drawing a line is not setting individual pixels. Just to be clear, the attached image is what I mean by drawing a line:

Thank you,

-Chad

[Chad Stearns]

Hi Floh,

Hey thats awesome! I had fun playing around with that for a little bit. I downloaded the code and studied it quite a bit. I was astonished by how little javascript was behind that application. I notice there is no 'yakcapp.js' in your repo, but I presume its built by your python files, and whatever cmake is (I tried to build your application locally, but I had some trouble).

How is there that much C++ in a web application? Does all the C++ get run by web-gl? I understand what you mean by RGBA8 format, and how you are storing the canvas. Am I right to believe that the actual business of modifying that array of pixels is in C++, and not JavaScript?

Thank you so much,

-Chad

[Jaume Sánchez]

Hi!

I think the best option for what you're describing is to use ImageData (https://developer.mozilla.org/en-US/docs/Web/API/ImageData). 

ImageData lets you address a section or the full canvas via via an Uint8ClampedArray, which is the closest you'll get to the memory representation of the image. You can then address individual pixels, or individual components per pixel (RGBA), copy sections, move data around, put back into the canvas. It doesn't deal with WebGL (it's all the Canvas 2D API), but it does have certain native acceleration (via Typed Arrays implementations and probably the default hardware acceleration that the browser might provide for canvas)

I hope it helps.

--
You received this message because you are subscribed to the Google Groups "WebGL Dev List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to webgl-dev-list+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Theo Armour]

Chad

I'm with Jaume

ImageData is a ton of fun. And you get a pixelator of sorts built in!

https://jsfiddle.net/theo/2f3gwpd4/

Theo

[Andre Weissflog]

Yes, this is a C++ application cross-compiled with emscripten, so everything interesting is happening in that yakcapp.js blob, sorry about that :) The little JScode in the git repo is just to integrate the asm.js blob with the rest of the webpage.

All WebGL rendering happens from C++ code (emscripten has a wrapper which translates call to the GL/GLES2 C-APIs to WebGL).

With RGBA8 I mean that the pixel format is Red/Green/Blue/Alpha with 8-bits per channel (so 4 bytes per pixel). Pixels are essentially written by the CPU into an UInt8 typed-array and the entire backing buffer is copied per frame into a WebGL texture. Although in the emulator this is all done in C++ it would work similar in Javascript, and would be about as fast.

Another advantage of this approach (apart from fragment shader image-effects) is that one can combine this with other WebGL rendering (such as the overlay-UI in the emulator).

Cheers,

-Floh.