Gamma, Colorspaces, Depth

[Dean McNamee]

I was having a quick look at using an increased bit depth in some
color blending intensive situations, where I thought the increased
precision could improve the results.

Having a look at SkImageInfo made me think that switching to F16 might
mean some other things in the system change also. Is there anywhere
that documents the colorspace/depth/etc supported by Skia and what the
differences are? For example a read of
SkColorAndColorSpaceAreGammaCorrect suggests that every F16 surface
will be gamma corrected. What exactly does that mean internally? If
a surface is gamma corrected, does operations (blending, etc) do an
implicit linearization/gamma correction? When are colorspaces
involved?

Also, what does this mean for reading in / getting out pixels? There
is a bug mentioning this:

https://bugs.chromium.org/p/skia/issues/detail?id=5022

Is also looks like there was a move from SkColorProfileType to
SkColorSpace, but SkYUVColorSpace is still on its own. I guess YUV is
mostly for JPEG? Why is this separate from the other colorspace
handling?

Thanks!

[Mike Reed]

Advanced color/buffers are still a work-in-progress, but I can give some answers now.

1. More precision

Skia is adding half-float buffers (16bit floats), both for Surfaces and Images.

2. Gamma correct

Gamma correct for Skia means that 32bit color values (e.g. SkColor and Images) will be "linearized" before blending/interpolating. To trigger this new-work-in-progress behavior, you need to

- tag your Surface with a SkColorSpace object (see SkColorSpace.h)

- set the SkSurfaceProps::kGammaCorrect_Flag flag

3. ColorSpace

SkColorSpace is our new type to identify the gamma and gamut of a surface/image. (SkColor, when in this new world, is treated as sRGB). Some practical constraints:

- Surfaces only support linear or sRGB gamma

- Encoded images (e.g. PNG, JPEG) can be stored using any gamma (via ICC Profiles)

- Drawing images (post-decode or provided explicitly) only support linear or sRGB gamma (like surfaces)

Gamut mapping today is only applied during image decode, but we expect that to be active during rendering as well (in the future).

4. SkYUVColorSpace

This enum is handy for specialized handling of video frames, and so is separate from the general-purpose SkColorSpace (at least for now).

--
You received this message because you are subscribed to the Google Groups "skia-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skia-discuss...@googlegroups.com.
To post to this group, send email to skia-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/skia-discuss.
For more options, visit https://groups.google.com/d/optout.

[Dean McNamee]

Thanks!

So is F16 half-float (which is more of 10-bit then), or is it unsigned
short 16-bit?

[Mike Reed]

half-float

[Mike Klein]

This prodded me to think about how many bits of precision our half floats actually hold.

Typically we use component values between 0 and 1 inclusively.  Lucky us, we get two zeros, 0x0000 and 0x8000.  Let's ignore them.

The 1023 other denormalized half floats carry 10 bits of mantissa, and the 14,337 normalized half floats less than or equal to 1 represent 11.  That's (10 * 1023 + 11 * 14337) / (1023 + 14337) ≈ 10.93 bits of mantissa per half float.  That's if you take the half floats uniformly... if you take the space between 0 and 1 uniformly, the denormalized values cover [0, 6.10352e-05) and normalized [6.10352e-05, 1], which puts the expected mantissa bits per half float at 10 * 6.10352e-05 + 11 * (1-6.10352e-05) ≈ 10.99994.

I suggest these results should lead us to conclude that half floats have approximately 11 bits of precision.

Now, allow me to extol some other virtues of half floats:

• they're a commonly supported GPU format, especially on mobile
• we can possibly do some neat things with component values <0 or >1, including -inf and +inf.
• both ARM and x86 now tend to ship with f16 <-> f32 conversion instructions
• ARM v8.2 will ship native half float math instructions
  

If we're not worried about GPU format compatibility, there's probably another weird format to consider for linear values between 0 and 1 before trying 16-bit fixed point: negative 15-bit fixed point.  By mapping 0.0 to 0 and 1.0 to -32768, we can use instructions like pmulhrsw on x86 or vqrdmulh on ARM (yep, real instructions, not cat-on-keyboard) to get full-bandwidth multiplies with proper rounding and renormalization.  Most other fixed-point formats cut your multiply bandwidth in half, as you need to temporarily represent the k*k = 2k bit results before renormalizing them back to k bits.

[Dean McNamee]

Thanks for the details. I didn't mean to particularly criticize
half-floats, I realized they were probably chosen with the GPU backend
in mind. I just immediately think of something else when I think of
'16-bit per pixel color', as in TIFF, etc.

Anyway, I think for what I'm doing the difference between 8 and 10
bits isn't really worth it. I would slightly argue with your 11-bit
calculation, not because the numbers don't exist but because the ULP
spacing is not uniform, you only have 10 bits of exponent=2^-1 range
(0.5 to 1), and then you get more as the numbers halve, because the
exponent shifts, and then a bunch more when you get to denorm but
these are tiny numbers which are useless for pixels. I don't see the
exponent-ness of floating point as a particularly good fit for linear
(or even gamma corrected) RGB, I would think you more or less always
want equal spacing. Also, not sure about GPUs but denorm floats are
usually a massive performance hit on x86 hardware.

Thanks again,
Dean

[Jim Van Verth]

I don't think GPUs support denorm floats. And in many cases (high precision variables, e.g.) fp16 is used as a storage format and converted to single-precision.

And it seems to me that floating point would be a good match for linear colors. You get more precision near zero, i.e., you get more distinction between blacks, which is what you want.

[Mike Klein]

I take it GPUs treat denorm floats as zero?  That's what ARMv7 NEON units do, and I really wish it were the default generally (as opposed to mysteriously running ~50x slower).  I assume the same is true for half floats, but out of curiosity have you ever checked?  After all, all numbers represented by denorm half floats can be represented by a normal float.

[Jim Van Verth]

Actually, I think I'm at least partially wrong. After the advent of CUDA and scientific computing on GPUs, I suspect the desktop GPUs do support denormals. But yes, if not I think they treat them as 0.

You've got me curious about the fp16->float conversion -- I'm not sure what they do there.

[Dean McNamee]

On Tue, Jul 12, 2016 at 4:49 PM, 'Jim Van Verth' via skia-discuss
<skia-d...@googlegroups.com> wrote:
> I don't think GPUs support denorm floats. And in many cases (high precision
> variables, e.g.) fp16 is used as a storage format and converted to
> single-precision.
>
> And it seems to me that floating point would be a good match for linear
> colors. You get more precision near zero, i.e., you get more distinction
> between blacks, which is what you want.

If I understood the Skia header properly F16 is stored gamma corrected
and not linear. Anyway, my point was actually I would prefer more
precision everywhere, not just towards zero. It depends what you are
doing, simply for storing pixels you want more information in the
blacks (which is what the gamma is doing), but for actually doing
blending and other operations I would prefer more precision
everywhere, including as colors mix towards white. If I understand
the F16 format is almost a bit like double gamma precision-wise,
because it's stored gamma corrected (giving more precision in blacks),
and then additionally the floating point format is giving you yet
again more precision in the blacks?

Thanks again,

[Mike Klein]

F16 is stored linear, gamma 1.0.

[Dean McNamee]

Okay, I must have misunderstood the meaning of "gamma correct" (I was
thinking it as gamma corrected):

// from SkImageInfo.h
static inline bool SkColorAndColorSpaceAreGammaCorrect(SkColorType ct,
SkColorSpace* cs) {
// Anything with a color-space attached is gamma-correct, as is F16.
// To get legacy behavior, you need to ask for non-F16, with a
nullptr color space.
return (cs != nullptr) || kRGBA_F16_SkColorType == ct;
}

On Wed, Jul 13, 2016 at 2:45 PM, 'Mike Klein' via skia-discuss

[Hal Canary]

Does that mean has-been-corrected, not needs-to-be-corrected?

[Brian Osman]

It means that we want to do linear math when the destination surface is one of those types. It actually says nothing about the gamma of the surface (which is 1.0 for F16, and ~2.2 for 8-bit sRGB), but about whether or not we apply the sRGB -> Linear conversion to inputs before operating on them in the shader (and blending them with the framebuffer). This is in contrast to our "legacy" 8-bit mode which was "gamma-oblivious", and blindly added/blended numbers that were sRGB encoded.

This is still the bleeding edge of feature development, and the exact terminology and API is still somewhat in flux. In particular, note the following caveats to what has already been said on this thread:

- The raster backend currently approximates sRGB gamma as 2.0, while the GPU backend uses the hardware sRGB support (piecewise function approximately 2.2).

- Raster always does gamma decoding of inputs based on the destination surface type (F16 and sRGB8888). GPU actually has a separate flag on SkSurfaceProps (kGammaCorrect_Flag) that controls that behavior. The helper function you found is just something that's used in our tools to decide if we should set that flag, so that we always perform gamma-correct rendering for those destination types on the GPU.

  - As a corollary, if you have a legacy application that is currently gamma-oblivious, and just want more precision without fixing anything else, you can use the GPU backend with F16, but not set that flag.

[Mike Klein]

This is gamma-correct in the sense of, if you ask Skia to perform an operation on this buffer that transforms linear inputs into linear outputs, the operation will be performed correctly.  This stands in contrast to a format like our status quo N32, which is typically sRGB encoded and interpreted as such by the display, but manipulated by Skia as if it were linear.  Anything we do with an N32 buffer beyond memcpy will produce mathematically incorrect results.  This is probably most visible in places where we linearly interpolate, like gradients and scaling.

[Mike Klein]

> The raster backend currently approximates sRGB gamma as 2.0

This is something we know we need to fix.  I hope to covert it all over to perfect sRGB -> linear and darn-good linear -> sRGB  Real Soon Now.

[Dean McNamee]

Okay, pretty clear now, sorry for the confusion and thanks for the answers.

On Wed, Jul 13, 2016 at 3:07 PM, 'Mike Klein' via skia-discuss