[whatwg] [canvas] request for {create, get, put}ImageDataHD and ctx.backingStorePixelRatio
Edward O'Connor
Unfortunately, lots of <canvas> content (especially content which calls
{create,get,put}ImageData methods) assumes that the <canvas>'s backing
store pixels correspond 1:1 to CSS pixels, even though the spec has been
written to allow for the backing store to be at a different scale
factor.
Especially problematic is that developers have to round trip image data
through a <canvas> in order to detect that a different scale factor is
being used.
I'd like to propose the addition of a backingStorePixelRatio property to
the 2D context object. Just as window.devicePixelRatio expresses the
ratio of device pixels to CSS pixels, ctx.backingStorePixelRatio would
express the ratio of backing store pixels to CSS pixels. This allows
developers to easily branch to handle different backing store scale
factors.
Additionally, I think the existing {create,get,put}ImageData API needs
to be defined to be in terms of CSS pixels, since that's what existing
content assumes. I propose the addition of a new set of methods for
working directly with backing store image data. (New methods are easier
to feature detect than adding optional arguments to the existing
methods.) At the moment I'm calling these {create,get,put}ImageDataHD,
but I'm not wedded to the names. (Nor do I want to bikeshed them.)
Thanks for your consideration,
Ted
Tab Atkins Jr.
Given that the modern iPhones (and I suspect the iPad 3, though I
haven't tested it yet) aren't exposing their high-res backing stores
(they give back ImageData with CSS px resolution), it seems likely
that the original goal of get/putImageData to seamlessly adapt has
failed. So, I support adding an alternate API that explicitly returns
a high-res store. If people fuck *that* up, then we're just screwed.
I'm not as sure about the backingStorePixelRatio bit. What's the
use-case for it? Why do devs need to detect this, and what will they
do different in the multiple code paths?
~TJ
James Robinson
make them asynchronous? This would allow for many optimization
opportunities that are currently difficult or impossible.
- James
Boris Zbarsky
> If we are adding new APIs for manipulating the backing directly, can we
> make them asynchronous? This would allow for many optimization
> opportunities that are currently difficult or impossible.
You mean like not blocking the world on the readback?
That would indeed be very nice. The question is what happens if drawing
happens after the getImageData call... Or for that matter after the
putImageData call (though I suspect there's less need for putImageData
to be async).
-Boris
Charles Pritchard
> Given that the modern iPhones (and I suspect the iPad 3, though I
> haven't tested it yet) aren't exposing their high-res backing stores
> (they give back ImageData with CSS px resolution), it seems likely
> that the original goal of get/putImageData to seamlessly adapt has
> failed. So, I support adding an alternate API that explicitly returns
IE exposes CSS px resolution via window.screen; mobile exposes via
devicePixelRatio, webkit exposes accidentally through innerHeight and
outerHeight and Mozilla through CSS queries on device-pixel-ratio.
I tried to get this mess fixed, but I got a lot of push back from Mozilla.
WebKit developers agreed that the MS solution would be OK:
http://msdn.microsoft.com/en-us/library/ms535868(v=vs.85).aspx
<http://msdn.microsoft.com/en-us/library/ms535868%28v=vs.85%29.aspx>
Anne agreed to add it to CSSOM back when he was editing it, once there
was a second implementation.
I've got two hacks to get pixel resolution over here:
http://www.jumis.com/cme-button.html#abc6
I didn't add the Mozilla one yet.
I strongly suggest we just fix the problem by updating window.screen so
us developers can manually manage the CSS width vs independent width.
I've been doing this forever, it works fine [I update on resize events
if the res has changed]:
<canvas style="width: 50px; height: 50px;" width="100" height="100" />
We already went through this discussion on WHATWG. I didn't like how it
went back then. Now that we're revisiting it, maybe we can just follow MS.
On desktop, the res changes with browser zoom.
-Charles
Charles Pritchard
I recommend we complete+use RoC's media processing API in addition to
the CSS shaders proposal:
http://www.w3.org/TR/streamproc/
https://dvcs.w3.org/hg/FXTF/raw-file/tip/custom/index.html
This would allow async post-processing via workers and less worry about
putImage semantics.
If we're looking for async getImageData purely for recognition, I think
the current postMessage transfer semantics sped things up enough.
getImageData and a subsequent draw call are always going to need to grab
more memory. async isn't going to change that.
-Charles
Edward O'Connor
> So, I support adding an alternate API that explicitly returns a
> high-res store. If people fuck *that* up, then we're just screwed.
Yup.
> I'm not as sure about the backingStorePixelRatio bit. What's the
> use-case for it? Why do devs need to detect this, and what will they
> do different in the multiple code paths?
Suppose you're a clever developer who basically does something like this
to handle both an iPhone 3GS and an iPhone 4:
if (window.devicePixelRatio == 1) {
// create a 100x100 canvas
} else if (window.devicePixelRatio == 2) {
// create a 200x200 canvas and scale it down to 100x100 with CSS
} // etc.
But now run through this logic when the <canvas> is making a high res
backing store automatically: by doing the clever thing, you're now
quadrupling the size of the canvas, and you're paying an exorbitant
storage cost for doing so.
You really only want to do the "make it twice as big and then scale it
down with CSS" trick when backing store pixels are 1:1 to CSS pixels.
Ted
Charles Pritchard
> But now run through this logic when the <canvas> is making a high res
> backing store automatically: by doing the clever thing, you're now
> quadrupling the size of the canvas, and you're paying an exorbitant
> storage cost for doing so.
Which (a): never happens and (b) can be detected via 1x1 pixel canvas.
> You really only want to do the "make it twice as big and then scale it
> down with CSS" trick when backing store pixels are 1:1 to CSS pixels.
I do "tricks" to support browser zoom. They are increments; .5,.7, 1.1, 1.2, 1.3, etc.
-Charles
Edward O'Connor
>> But now run through this logic when the <canvas> is making a high res
>> backing store automatically: by doing the clever thing, you're now
>> quadrupling the size of the canvas, and you're paying an exorbitant
>> storage cost for doing so.
>
> Which (a): never happens
Sorry, what never happens? Developers commonly double the size of their
<canvas>es (and scale them down with CSS) to support both the iPhone 3GS
and iPhone 4. Which means such code would use 4 times as much memory as
intended when <canvas> uses such a backing store.
> and (b) can be detected via 1x1 pixel canvas.
Having to round-trip image data through a <canvas> in order to detect
its backing store size is one of the problems I'm trying to solve here.
>> You really only want to do the "make it twice as big and then scale
>> it down with CSS" trick when backing store pixels are 1:1 to CSS
>> pixels.
>
> I do "tricks" to support browser zoom. They are increments; .5,.7,
> 1.1, 1.2, 1.3, etc.
Huh? I'm not sure what you mean by "browser zoom," nor do I know what it
has to do with my proposed additions to the <canvas> 2D Context API.
Ted
Charles Pritchard
> Charles Pritchard wrote:
>
>>> But now run through this logic when the <canvas> is making a high res
>>> backing store automatically: by doing the clever thing, you're now
>>> quadrupling the size of the canvas, and you're paying an exorbitant
>>> storage cost for doing so.
>>
>> Which (a): never happens
>
> Sorry, what never happens?
The backing store itself is never set by 2x in the implementation. Not in any public implementations I've seen. It's always 1:1 with height and width units.
> Developers commonly double the size of their
> <canvas>es (and scale them down with CSS) to support both the iPhone 3GS
> and iPhone 4. Which means such code would use 4 times as much memory as
> intended when <canvas> uses such a backing store.
It would do that, but it doesn't, because none of the implementations use a larger backing store.
And yes, in preparing for some future break in implementations, it may be wise for authors to run a check on getImageData. I don't, I've not seen it done, but it may be prudent.
>
>> and (b) can be detected via 1x1 pixel canvas.
>
> Having to round-trip image data through a <canvas> in order to detect
> its backing store size is one of the problems I'm trying to solve here.
It's a 1:1 fetch, it doesn't take any time to do getImageData(0,0,1,1);
I don't mean to be pushing back on this issue. I'm more focused on the fact that other obvious issues have not been addressed by vendors, and now we're examining an issue that does not yet exist.
>
>>> You really only want to do the "make it twice as big and then scale
>>> it down with CSS" trick when backing store pixels are 1:1 to CSS
>>> pixels.
>>
>> I do "tricks" to support browser zoom. They are increments; .5,.7,
>> 1.1, 1.2, 1.3, etc.
>
> Huh? I'm not sure what you mean by "browser zoom," nor do I know what it
> has to do with my proposed additions to the <canvas> 2D Context API.
I know, and it'd be swell if we could sit down some time and I could walk you through the Canvas API an WCAG.
In the meantime, perhaps you could go over to the link I sent (A canvas button) on Jumis.com.
By browser zoom, I mean on a desktop, when I use CTRL + to change the pixel ratio on the page.
I've posted links to MS docs, and to a live example, on this thread.
-Charles
Glenn Maynard
> That would indeed be very nice. The question is what happens if drawing
> happens after the getImageData call... Or for that matter after the
> putImageData call (though I suspect there's less need for putImageData to
> be async).
>
The drawing calls that happen after would need to be buffered (or otherwise
flush the queue, akin to calling glFinish), so the operations still happen
in order.
putImageData being async makes sense, too, for the same reason: it avoids
having to flush drawing commands earlier in the queue, which helps keep
putImageData from blocking. It's a bit trickier, though: what happens if
the argument passed to putImageData is modified before it's written? You'd
either need a mechanism to detect changes, so you can make a copy (eg. a
copy-on-write mechanism for ArrayBuffer--though that sort of sounds useful
in its own right), or to just say that any changes to made to the buffer
before the async operation completes will be reflected in the copy.
--
Glenn Maynard
Charles Pritchard
> On Tue, Mar 20, 2012 at 2:08 PM, Boris Zbarsky<bzba...@mit.edu> wrote:
>
>> That would indeed be very nice. The question is what happens if drawing
>> happens after the getImageData call... Or for that matter after the
>> putImageData call (though I suspect there's less need for putImageData to
>> be async).
>>
> The drawing calls that happen after would need to be buffered (or otherwise
> flush the queue, akin to calling glFinish), so the operations still happen
> in order.
Webkit did land buffered drawing operations.
When working with Flash (way back when) as a Canvas polyfill, buffered
drawing made a huge difference. I doubt it has much of a performance
impact now, except when rendering is done on some high latency pipeline
(such as, perhaps, the GPU).
The frustrating item here; the area where there may be a clear
optimization or win is with video/webcam.
We have to do: drawImage(video).getImageData() for each frame [of
interest]. That one would be nice to have optimized.
We can't use RoC's media stream processing API (workers) because it's
for output, not input. We don't need the canvas to keep a copy of the
image in buffer after the getImageData call.
Beyond that case though, I doubt there's much to be done here
-Charles
Boris Zbarsky
> The drawing calls that happen after would need to be buffered (or
> otherwise flush the queue, akin to calling glFinish), so the operations
> still happen in order.
The former seems like it could get pretty expensive and the latter would
negate the benefits of making it async, imo.
> putImageData being async makes sense, too, for the same reason: it
> avoids having to flush drawing commands earlier in the queue, which
> helps keep putImageData from blocking.
I don't see why it needs to block at all. At least in Gecko the
putImageData basically just becomes a drawing command itself; you send
it over to the graphics card and forget about it.
> what happens if the argument passed to putImageData is modified before
> it's written?
You have to copy it, yes. Which you may have to do anyway, because
imagedata is not premultiplied and for most drawing you want
premultiplied data.
> You'd either need a mechanism to detect changes, so you
> can make a copy (eg. a copy-on-write mechanism for ArrayBuffer--though
> that sort of sounds useful in its own right), or to just say that any
> changes to made to the buffer before the async operation completes will
> be reflected in the copy.
That seems unfortunately racy. Also unnecessary, imo.
-Boris
Glenn Maynard
> On 3/20/12 6:36 PM, Glenn Maynard wrote:
>
>> The drawing calls that happen after would need to be buffered (or
>> otherwise flush the queue, akin to calling glFinish), so the operations
>> still happen in order.
>>
>
> The former seems like it could get pretty expensive and the latter would
> negate the benefits of making it async, imo.
The latter just means that implementations aren't *required* to actually
buffer drawing operations.
It sounds like implementations are already doing the former, or want to,
from what James said. It's not inherently expensive, especially if the
input parameters to the drawing call are lightweight, which most canvas
calls are. OpenGL has always buffered commands like this. By buffering
the calls, you can push the actual drawing off to a thread and avoid
blocking the UI thread.
I don't see why it needs to block at all. At least in Gecko the
> putImageData basically just becomes a drawing command itself; you send it
> over to the graphics card and forget about it.
If you have previous drawing commands buffered, and you want to avoid extra
copies, then putImageData has to block until the buffered drawing commands
complete.
Avoiding that extra copy may not be worth the complexity, though.
what happens if the argument passed to putImageData is modified before
>> it's written?
>>
>
> You have to copy it, yes. Which you may have to do anyway, because
> imagedata is not premultiplied and for most drawing you want premultiplied
> data.
The question is whether you'd need to make a copy *synchronously*, before
putImageData returns. Manipulating the data you put into the image doesn't
have to happen until the actual blit occurs (and the two may happen in the
same pass).
--
Glenn Maynard
Boris Zbarsky
> If you have previous drawing commands buffered, and you want to avoid extra
> copies, then putImageData has to block until the buffered drawing commands
> complete.
Yes, but if you're drawing to a GPU directly you want to make the copy
up front, imo; otherwise you have to wait for the full GPU latency
before you can return even if there are no other drawing commands in the
pipeline, which is painful....
> The question is whether you'd need to make a copy *synchronously*, before
> putImageData returns.
If you want to do the image data put async in any way (and that includes
any sort of direct-to-GPU setup, I'm told) then you need either sync
copy or copy-on-write as far as I can tell.
-Boris
Maciej Stachowiak
On Mar 20, 2012, at 3:22 PM, Charles Pritchard wrote:
> On Mar 20, 2012, at 3:05 PM, Edward O'Connor <eoco...@apple.com> wrote:
>
>> Charles Pritchard wrote:
>>
>>>> But now run through this logic when the <canvas> is making a high res
>>>> backing store automatically: by doing the clever thing, you're now
>>>> quadrupling the size of the canvas, and you're paying an exorbitant
>>>> storage cost for doing so.
>>>
>>> Which (a): never happens
>>
>> Sorry, what never happens?
>
> The backing store itself is never set by 2x in the implementation. Not in any public implementations I've seen. It's always 1:1 with height and width units.
We're considering the possibility of scaling the backing store in future releases (which we can't really discuss in detail). We have experimented with it in WebKit, and we believe it's not viable to ship a production browser with backing store scaling without the sorts of API changes that Ted proposed because of how much content breaks.
An automatically scaled backing store is better for authors, because for the case where they are not doing any direct pixel manipulation, they get higher quality visual results with no code changes on devices that scale CSS pixels. But to offer it, we need to take care of the compatibility issues, and also provide a path for authors who have gone the extra mile to hand-scale 1x backing stores on 2x devices. In other words, all the following cases need to work:
devicePixelRatio is 1; backingStorePixelRatio is 1.
devicePixelRatio is 2; backingStorePixelRatio is 1.
devicePixelRatio is 2; backingStorePixelRatio is 2.
Maybe even other possibilities. In other words, we don't want to force either the assumption that backingStorePixelRatio is always 1, or that it is always is equal to devicePixelRatio. We believe that in time, neither is a safe assumption.
Regards,
Maciej
Maciej Stachowiak
On Mar 21, 2012, at 8:31 PM, Charles Pritchard wrote:
> On 3/21/2012 8:21 PM, Maciej Stachowiak wrote:
>> On Mar 20, 2012, at 3:22 PM, Charles Pritchard wrote:
>>
>>> On Mar 20, 2012, at 3:05 PM, Edward O'Connor<eoco...@apple.com> wrote:
>>>
>>>> Charles Pritchard wrote:
>>>>
>>>>>> But now run through this logic when the<canvas> is making a high res
>>>>>> backing store automatically: by doing the clever thing, you're now
>>>>>> quadrupling the size of the canvas, and you're paying an exorbitant
>>>>>> storage cost for doing so.
>>>>> Which (a): never happens
>>>> Sorry, what never happens?
>>> The backing store itself is never set by 2x in the implementation. Not in any public implementations I've seen. It's always 1:1 with height and width units.
>> We're considering the possibility of scaling the backing store in future releases (which we can't really discuss in detail). We have experimented with it in WebKit, and we believe it's not viable to ship a production browser with backing store scaling without the sorts of API changes that Ted proposed because of how much content breaks.
>
> The change being the addition of a "backingStorePixelRatio" or the change being the addition of a second set of "HD" items?
We think both those changes are required to handle all cases gracefully.
>
> I get what you're saying about HD; if the user requests a non-HD, it'd return a typical 1:1 backing store, which most sites expect.
> Still, it seems a bit weird.
>
> Why not use the method that already exists of managing the CSS and devicePixelRatio? If an author is using new methods,
> they're certainly able to use the old ones.
I'm not sure what you mean by that. As I mentioned, backingStorePixelRatio is in general not equal to devicePixelRatio. It's true that you might be able to infer the backing store scale by creating a canvas solely for testing, but that is needlessly awkward.
>
>
>> An automatically scaled backing store is better for authors, because for the case where they are not doing any direct pixel manipulation, they get higher quality visual results with no code changes on devices that scale CSS pixels. But to offer it, we need to take care of the compatibility issues, and also provide a path for authors who have gone the extra mile to hand-scale 1x backing stores on 2x devices. In other words, all the following cases need to work:
>>
>> devicePixelRatio is 1; backingStorePixelRatio is 1.
>> devicePixelRatio is 2; backingStorePixelRatio is 1.
>> devicePixelRatio is 2; backingStorePixelRatio is 2.
>>
>> Maybe even other possibilities. In other words, we don't want to force either the assumption that backingStorePixelRatio is always 1, or that it is always is equal to devicePixelRatio. We believe that in time, neither is a safe assumption.
>>
>
> Well if they --need-- to work, better to add the value sooner than later.
>
> My concern is that you've also got window.screen.logicalXPixelRatio on the desktop.
>
> You'll really have three items now to add up.
>
> devicePixelRatio * backingStorePixelRatio * logicalPixelRatio.
>
> Is that middle item really necessary?
> I wasn't able to get anyone to budge on changing window.devicePixelRatio on the desktop. It's fixed at 1.
I was unable to decipher what IE's logical{X,Y}DPI does and how it differs from device{X,Y}DPI and for that matter system{X,Y}DPI. But I don't believe any of those things relate to the canvas backing store, however, so I don't see how they eliminate the need for backingStoreRatio.
Regards,
Maciej
Charles Pritchard
>> You'll really have three items now to add up.
>>
>> devicePixelRatio * backingStorePixelRatio * logicalPixelRatio.
>>
>> Is that middle item really necessary?
>> I wasn't able to get anyone to budge on changing window.devicePixelRatio on the desktop. It's fixed at 1.
>
> I was unable to decipher what IE's logical{X,Y}DPI does and how it differs from device{X,Y}DPI and for that matter system{X,Y}DPI. But I don't believe any of those things relate to the canvas backing store, however, so I don't see how they eliminate the need for backingStoreRatio.
When you zoom out or in on a page, the ratio changes. So if I check that value after a resize event I know to change the units on my canvas elements if I want them to not be blurry (when zoomed in) or if I want to not do excess work (when zoomed out).
What is the benefit of drawing to an over sized ("bush res") backing store? Seems like on a device where zoom is very common (yes, you, iphone), it could make for a little nicer experience. On desktop though, we just repaint on resize and zoom transitions are scaled through the gpu anyway.
Seems like a lot of extra work for the phone though. And we can do it as authors by just using CSS width = .5* width;
I agree with your assessment, both features are necessary to bring it in.
I'd still like someone o'er in WebKit to pick up the issue that's existed since the introduction of Canvas in WebKit: exposing the current pixel ratio so we can redraw our Canvas at the appropriate ratio when browser zoom (zoom in or out) is in use.
Currently I do outerWidth/innerWidth to estimate.
-Charles