Intent to implement: image-rendering: pixelated

[Jack Hou]

Primary eng (and PM) emails

jac...@chromium.org

Spec

http://dev.w3.org/csswg/css-images/#the-image-rendering

Summary

Implement the "image-rendering: pixelated" property. This indicates that the algorithm used to scale the image should be nearest-neighbor.

Motivation

This enables high performance UI for image editing applications. In order to support zooming, these applications will typically redraw the image data to another canvas at the required scale. This involves either redrawing the entire image frequently or hooking into scroll events to redraw parts of the image, both of which are prone to jank. Adding this feature will improve performance and decrease complexity and memory usage for this class of applications.

Compatibility Risk

This is supported on other browsers in the following forms:

Firefox: "image-rendering: -moz-crisp-edges"

Webkit: "image-rendering: -webkit-optimize-contrast"

IE: "-ms-interpolation-mode: nearest-neighbor"

Ongoing technical constraints

None.

Will this feature be supported on all five Blink platforms (Windows, Mac, Linux, Chrome OS and Android)?

Yes.

OWP launch tracking bug?

None

Row on feature dashboard?

No

Requesting approval to ship?

No

Other

I'm not familiar with working in blink, so it would be ideal if someone with more experience in this area is interested in taking this on. That said, I'm happy to implement it if necessary.

[Adam Barth]

Do we support -webkit-optimize-contrast?

[Jack Hou]

Not at the moment. My understanding from is that it was implemented for webkit but does not work with skia.

[Darin Fisher]

Have any other browser vendors shown support for implementing "image-rendering: pixelated"? It is striking that no one else calls it that.

-Darin

On Mon, Feb 24, 2014 at 9:51 PM, Jack Hou <jac...@chromium.org> wrote:

[Jack Hou]

It was proposed in CSS3 with the values 'optimizeSpeed' and 'optimizeQuality', but was moved to CSS4 with new values 'auto', 'crisp-edges', and 'pixelated'.

Webkit went with 'optimizeQuality', '-webkit-crisp-edges', and '-webkit-optimize-contrast':

http://trac.webkit.org/changeset/148949/trunk/Source/WebCore/css/CSSValueKeywords.in

Firefox has a bug for supporting 'pixelated':

https://bugzilla.mozilla.org/show_bug.cgi?id=856337

But it seems from the documentation that '-moz-crisp-edges' does the same thing:

https://developer.mozilla.org/en/docs/Web/CSS/image-rendering

I'm not fussed about what it's called. It probably makes sense to put it behind a prefix until it's finalized.

[Adam Barth]

On Mon, Feb 24, 2014 at 11:58 PM, Jack Hou <jac...@google.com> wrote:

It was proposed in CSS3 with the values 'optimizeSpeed' and 'optimizeQuality', but was moved to CSS4 with new values 'auto', 'crisp-edges', and 'pixelated'.

Webkit went with 'optimizeQuality', '-webkit-crisp-edges', and '-webkit-optimize-contrast':

http://trac.webkit.org/changeset/148949/trunk/Source/WebCore/css/CSSValueKeywords.in

Firefox has a bug for supporting 'pixelated':

https://bugzilla.mozilla.org/show_bug.cgi?id=856337

But it seems from the documentation that '-moz-crisp-edges' does the same thing:

https://developer.mozilla.org/en/docs/Web/CSS/image-rendering

I'm not fussed about what it's called. It probably makes sense to put it behind a prefix until it's finalized.

We don't add prefixed APIs anymore.  We can keep it behind a runtime flag until it's ready, though.

Adam

[Stephen Chenney]

+skia-discuss

[Mike Reed]

Is this purely a "how should the HTML or CSS specify this" question, or is there also a question about how to support this at the graphics layer?

--
You received this message because you are subscribed to the Google Groups "skia-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skia-discuss...@googlegroups.com.
To post to this group, send email to skia-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/skia-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

[Dirk Schulze]

On Feb 25, 2014, at 8:58 AM, Jack Hou <jac...@google.com> wrote:

> It was proposed in CSS3 with the values 'optimizeSpeed' and 'optimizeQuality', but was moved to CSS4 with new values 'auto', 'crisp-edges', and 'pixelated’.

optimizeSpeed and optimizeQuality are used by SVG and should be implemented in most browsers one way or the other. IIRC CSS Images 4 still makes them mandatory.

Greetings,
Dirk

PS: ‘pixelation' sounds a bit strange, I like 'crisp-edges’ better which should describe it better.

[Sami Kyostila]

Is this purely for <img> or will it work with other elements such as <canvas>? (e.g., http://stackoverflow.com/questions/7615009/disable-interpolation-when-scaling-a-canvas)

- Sami

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Stephen Chenney]

The question for Skia is: Does skia already have a nearest neighbor filter resize operation? If so the graphics back end of this is easy, if not we'll need to do some coordinating.

[Stephen White]

When compositing the canvas itself, if it is resized, we should respect this property (whatever the name) for <canvas> as we do for other elements, such as images.

(Note that <canvas> already has support for nearest-neighbour when drawing images and other canvases into a canvas, via ctx.imageSmoothingEnabled = false, although that's not what the stackoverflow question is about.)

Stephen

[Mike Reed]

Resize options are all in SkPaint, and its default setting is nearest-neighbor. (kNone_FilterLevel)

[Rik Cabanier]

I think this is a useful feature but I wonder about your use case.

If you create an image editing application, wouldn't you want to draw your UI controls and selection areas non-pixelated?

In addition, Canvas is very fast at scrolling (see all the canvas games) so I don't understand why doing a drawImage with 'imageSmoothingEnabled = false' would cause jank.

On Mon, Feb 24, 2014 at 9:51 PM, Jack Hou <jac...@chromium.org> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Tab Atkins Jr.]

On Mon, Feb 24, 2014 at 11:58 PM, Jack Hou <jac...@google.com> wrote:

> It was proposed in CSS3 with the values 'optimizeSpeed' and
> 'optimizeQuality', but was moved to CSS4 with new values 'auto',
> 'crisp-edges', and 'pixelated'.
>
> Webkit went with 'optimizeQuality', '-webkit-crisp-edges', and
> '-webkit-optimize-contrast':
> http://trac.webkit.org/changeset/148949/trunk/Source/WebCore/css/CSSValueKeywords.in

Wow, that's a big bag of weird.

> Firefox has a bug for supporting 'pixelated':
> https://bugzilla.mozilla.org/show_bug.cgi?id=856337
> But it seems from the documentation that '-moz-crisp-edges' does the same
> thing:
> https://developer.mozilla.org/en/docs/Web/CSS/image-rendering

Having "crisp-edges" act like "pixelated" is an allowed implementation
strategy. It's also allowed, though, to be smarter when doing
"crisp-edges", and use an intelligent pixel-scaling algorithm, of
which there are many.

"pixelated" was added by request of multiple users, who sometimes
literally want the "big pixel" look of plain nearest-neighbor
interpolation.

~TJ

[Mike Lawther]

Some quick history: I implemented -webkit-optimize-contrast way back when. I made it work with CoreGraphics first, and then went on to skia. Skia already supports nearest neighbor filtering - there are a bunch of heuristics under the hood that decide what sort of filtering to use for performance reasons (eg if an image is moving - that was done to improve our performance on this demo: http://ie.microsoft.com/TestDrive/Performance/FlyingImages/Default.html)

I got stuck in several discussions: about Skia listening to WebKit about what filtering to use, and also whether WebKit should be explicit about what filtering algorithms to use. At the time WebKit had a 1:1 mapping to CoreGraphics' CGInterpolationQuality, which only specified Low, Medium, High etc, rather than eg NearestNeighbour, Bilinear, Bicubic etc. It so happened that CG used nearest neighbor for 'Low', but that was an implementation detail rather than a guarantee etc etc. Skia uses 'None' to mean nearest neighbor, and there was unhappiness about this 'Low means None in this case'. 

And then the image-rendering property got pulled from the Level 3 spec, so all appetite was lost, and this was backburnered until a Level 4 spec came out. Which I guess could be now.

There's data to show that using nearest neighbour is faster, and so giving the developer control is a good idea. With Blink only supporting a single backend, most of these old worries about supporting mapping between multiple engines should be moot.

[Justin Novosad]

FWIW, there is long-standing demand for this feature.  Implementing this will make several devs really happy.

Evidence:  crbug.com/106662 (tl;dr open since december 2011, starred by 69 people)

[Jack Hou]

On Wed, Feb 26, 2014 at 3:29 AM, Sami Kyostila <skyo...@google.com> wrote:

Is this purely for <img> or will it work with other elements such as <canvas>? (e.g., http://stackoverflow.com/questions/7615009/disable-interpolation-when-scaling-a-canvas)

Thanks for pointing this out. I should have made it clear, this will work with <img> and <canvas>, since that's where there is most demand for this feature. 

[Jack Hou]

If you create an image editing application, wouldn't you want to draw your UI controls and selection areas non-pixelated?

In addition, Canvas is very fast at scrolling (see all the canvas games) so I don't understand why doing a drawImage with 'imageSmoothingEnabled = false' would cause jank.

It's more for displaying the image being edited. If you want to zoom in on the image, the editor needs to implement redrawing the visible part of the image whenever the bounds change. This can be avoided completely if the platform supports pixelated upscaling. Zoom would be a simple "transform: scale()", which you could also combine with web animations.

[Dirk Schulze]

Implementations usually just have the choice between nearest-neighbor, bilinear and bicubic interpolations. These terms are actually well known. I am not sure if ‘pixelated’ is really the best term to describe nearest-neighbor. As a consequence, crisp-deges is usually implemented with nearest-neigbors as well. I do not see a reason to keep both keywords. And I don’t think pixelation is the right term to use.

Greetings,
Dirk



>
> ~TJ

[Dirk Schulze]

Btw. I assume this will be behind a flag. It is just an "intent to implement" request and not to ship”.

Greetings,
Dirk

[Mike Lawther]

On 26 February 2014 17:11, Dirk Schulze <dsch...@chromium.org> wrote:

Implementations usually just have the choice between nearest-neighbor, bilinear and bicubic interpolations. These terms are actually well known. I am not sure if ‘pixelated’ is really the best term to describe nearest-neighbor. As a consequence, crisp-deges is usually implemented with nearest-neigbors as well. I do not see a reason to keep both keywords. And I don’t think pixelation is the right term to use.

At the risk of continuing a discussion best had on www-style, Tab has already covered why both keywords exist earlier in this thread. 'crisp-edges' is intended to allow scalers such as AdvMame2x or even ones like http://research.microsoft.com/en-us/um/people/kopf/pixelart/supplementary/comparison_super2xsai.html.

To be clear - this intent to implement is not about doing anything like the above for 'crisp-edges'. It is only about doing nearest-neighbor scaling with 'pixelated' (or whatever property value it ends up being). 

[Daniel Bratell]

This has probably been mentioned elsewhere too, but the word "pixelated"
seems to match the expected behaviour perfectly when zooming in (scaling
up) but it feels all wrong to call it "pixelated" when zooming out
(scaling down).

/Daniel

[Tab Atkins Jr.]

On Tue, Feb 25, 2014 at 10:11 PM, Dirk Schulze <dsch...@chromium.org> wrote:
> Implementations usually just have the choice between nearest-neighbor, bilinear and bicubic interpolations. These terms are actually well known. I am not sure if ‘pixelated’ is really the best term to describe nearest-neighbor. As a consequence, crisp-deges is usually implemented with nearest-neigbors as well. I do not see a reason to keep both keywords. And I don’t think pixelation is the right term to use.

The terms are well-known to anyone familiar with graphics terminology,
but not to the general public (including the vast majority of CSS
authors). So, I instead named the keyword after the effect people are
actually trying to achieve, which is "pixelated". This is consistent
with the other intent-based keywords.

~TJ

[Tab Atkins Jr.]

None of the keywords mean very much when downscaling, so I'm not too
concerned about that. ^_^

~TJ

[Michael Thiessen]

I personally find the choice of the keyword crisp-edges very strange. Most intelligent re-scaling algorithms (by that I mean non-nearest neighbor) don't preserve crisp edges, and produce blurred images. Also, another way nearest-neighbor is commonly referred to is simply 'resize'.

For the content shown here http://research.microsoft.com/en-us/um/people/kopf/pixelart/supplementary/comparison_super2xsai.html,
I would find the following keywords to be less confusing:
'resize' - refers to nearest-neighbor algorithm

'vectorize' - refers to the crisp-edged style scaling microsoft referred to as 'our method'

'resample' - refers to algorithms like 2xSaI

'auto' - whatever the default is

Cheers,

Michael

[Daniel Bratell]

On Fri, 28 Feb 2014 19:21:38 +0100, Tab Atkins Jr. <jacka...@gmail.com>
wrote:

Except that sometimes people just want performance and this is the fastest
algorithm, especially when downscaling, in which case "pixelated" doesn't
describe the graphical effect of the scaling operation at all.
"crisp-edged" would in that case work better (as it preserves individual
pixels and does not blur everything together).

/Daniel

[Dongseong Hwang]

The spec mention that _MUST_ use "nearest neighbor".

pixelated

When scaling the image up, the "nearest neighbor" or similar algorithm must be used, 

IMO, it's problematic. Let's say Accelerated2DCanvas and Accelerated Compositing. GPU can do bilinear scale very well.

Above restriction perhaps forces gpu code dirty without obvious performance gain.

DS

[Tab Atkins Jr.]

On Fri, Feb 28, 2014 at 11:04 AM, Michael Thiessen
<zenithan...@gmail.com> wrote:
> I personally find the choice of the keyword crisp-edges very strange. Most
> intelligent re-scaling algorithms (by that I mean non-nearest neighbor)
> don't preserve crisp edges, and produce blurred images.

Those algorithms are inappropriate to use for the 'crisp-edges' value,
since they violate its definition.

> Also, another way
> nearest-neighbor is commonly referred to is simply 'resize'.

I've never heard of this usage before.

> For the content shown here
> http://research.microsoft.com/en-us/um/people/kopf/pixelart/supplementary/comparison_super2xsai.html,
> I would find the following keywords to be less confusing:
> 'resize' - refers to nearest-neighbor algorithm
> 'vectorize' - refers to the crisp-edged style scaling microsoft referred to
> as 'our method'
> 'resample' - refers to algorithms like 2xSaI
> 'auto' - whatever the default is

Unless use-cases are presented, I'm against mandating specific
algorithms. The idea right now is to capture a general *intent*, and
let browsers trade off fidelity versus speed in satisfying that
intent.

~TJ

[Tab Atkins Jr.]

When downscaling, "pixelated" is equivalent to "auto".

~TJ

[Tab Atkins Jr.]

Bilinear scaling doesn't look anything like the "big pixels" effect
that people have requested. It's pretty terribly ugly if you're trying
to upscale pixel graphics.

The "pixelated" value is not a performance hack, it's a rendering
intent. You specify it when that's the specific look you're hoping
for.

~TJ

[Michael Thiessen]

Just to clarify, I wasn't intending to imply that the categories 'resize', 'vectorize', and 'resample' refer to specific algorithms. These categories are intended to represent the types of scaling algorithms, not the actual implementation. 'crisp-edges' and 'pixelated' don't allow for the use of many common scaling methods.

The names I suggest could well be changed, I don't intend to suggest that I know what the names should be, but I feel there should be three categories (based on the different types of visual results).

The algorithms like hqx, linear, bilinear could be implementations of a category referred to as 'resampled' or something else, where the intent is to essentially maintain the look of the image, but make it larger (use case would be photographs, etc)

The various vectorizing algorithms would fall under a category referred to as 'vectorized', where the intent is to make clean and sharp looking images at any scale (use cases could involve UI components like buttons, game sprites, etc.)

The 'pixelated' algorithms like nearest-neighbor could be called 'pixelated', or 'simple', or 'resize', or some other term (use case is pixelated look or speed)

I just find that the 'pixelated' and 'crisp-edges' don't cover common use cases, and I'm really not sure what scaling type 'crisp-edges' is referring to other than possibly pixelated, or maybe hqx, but even these don't create particularly 'crisp' edges. I feel that any user that cares or knows enough to specify how they want their image rendered will just be more confused by what 'crisp-edges' means (Which edges? What if the image has no well defined edges? If I'm upscaling a photograph, I don't want an algorithm introducing crisp edges, and I don't want my image looking pixelated, what do I do?).

~TJ

[dgs...@gmail.com]

Just pointing out that this feature might be useful for image-editing apps, but it is definitely essential for games, which usually have a smaller virtual resolution and gets blurry when scaled to bigger window sizes and full screen modes.

[speed...@gmail.com]

We have now septermber and still nothing has been done?
It is very desired feature by many for years.

There are many topics related to this problem over internet (some can be found on stackoverflow).
It was solved long time ago on FireFox with using "image-rendering: -moz-crisp-edges;" and works great today.

[PhistucK]

Why nothing? It was implemented and will be featured in the next release, Chrome 38 -

http://blog.chromium.org/2014/08/chrome-38-beta-new-primitives-for-next.html

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[speed...@gmail.com]

I am sorry, I didnt noticed it (using 37).
Thats great news and I am going to test it out and fix some pixel style games.

[Diogo Schneider]

Great news!

Still wondering why this is not the default behavior, though.

[Tab Atkins Jr.]

On Tue, Sep 2, 2014 at 5:08 AM, Diogo Schneider <dgs...@gmail.com> wrote:
> Still wondering why this is not the default behavior, though.

Because pixelated upscaling is strictly worse for photos and similar
"normal" pictures, which make up most of the images on the web. The
types of images that benefit from pixelated upscaling are relatively
rare, and so it's correct to make them require the opt-in.

~TJ

[speed...@gmail.com]

It was working great on Chrome 38.
It is no more working on Chrome 39.0.2171.13 beta-m (64-bit).

On Monday, September 1, 2014 10:23:53 PM UTC+2, PhistucK wrote:

[PhistucK]

Right, the recent temporary conclusion of the intent to ship thread was to disable the feature due to issues concerning hardware acceleration of images and canvas.

While it was indeed included in beta (up until recently, I guess), the final and stable Chrome 38 (out recently) does not have the feature.

☆PhistucK

[speed...@gmail.com]

Does it mean that this feature will not be included in near future updates of Chrome?
Not sure if similar problems exist in FireFox, but they support non interpolation for a long time now and it is working for me fine.

[PhistucK]

The bug is not fixed yet, as far as I know, so, no, it will not show up in the next release.

☆PhistucK