Custom SkBlitter to support user defined image formats

[D Marc]

Hello Skia Community!

I would like to use Skia to raster/draw to an image in a format that Skia doesn't support.

Looking at the code, I figured that I could do that if I implemented my own SkBlitter. I can use that to create a custom SkImageInfo, which again can be used to create a SkSurface by calling MakeRaster().

It turned out it was rather easy to modify the Skia library in this way and I was able to raster/draw to an image in a previously unsupported image format with my experimental snapshot.

I was wondering whether you could possibly consider to make the SkBlitter API public to allow making custom SkImageInfos allowing to support pretty much any image format.

I did something along the lines:

MyImage image = ...;

class MyBlitter : SkBlitter {...};

SkImageInfo info = SkImageInfo::MakeCustom(image.width(), image.height(), MyCreateBlitter, (void*)&image);

 sk_sp<SkSurface> surface = SkSurface::MakeRaster(info);

 SkCanvas* canvas = _surface->getCanvas();

where MyCreateBlitter is the factory function that creates the custom SkBlitter for SkImageInfo.

best regards,

Marc

[Brian Osman]

I don't think we want to allow custom SkBlitters like this, but I'm curious why you needed to go that far? The blitter itself encapsulates a huge amount of functionality that is unrelated to the color type used for the surface. Did you base your blitter on one of the existing ones? What are the properties of your format? (It might be much easier to simply support custom color types in SkRasterPipelineBlitter, for example).

--
You received this message because you are subscribed to the Google Groups "skia-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skia-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/f1296111-ba1c-4ebf-bb68-63ee887e8fefn%40googlegroups.com.

[D Marc]

Hello Brian,

Thank you for your reply!

In terms of the image format I want to be able to draw directly to
pixel based and planar formats in various color formats.

I based my blitter on SkARGB32_Blitter, which is rather simple, less
than 200 lines of code. As far as I understood it, SkBlitter receives
a color as part of a SkPaint and is responsible for setting pixels in
the target image in that color. So it needs to know how to convert the
color from SkPaint to the respective color format of the target image
and it needs to know how to access the pixels or planes in the target
image. I used an Image class which already abstracts pixel access and
color format, so it was pretty straightforward to adapt
SkARGB32_Blitter for that purpose. At this point it seems to me
generally useful to have a way to let Skia draw to image or buffers in
a format that Skia does not support.

Your idea of a custom color type certainly sounds promising too. How
would you do this? Are we talking about the SkColorType enum?

best regards,
Marc

> To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CAMcBjo4Ln6DeGeNYq5k_AkQr6VN21Rx%2BVf8BeYNVKTDJx9PQrQ%40mail.gmail.com.

[Brian Osman]

Hi Marc,

Unfortunately, SkARGB32_Blitter is extremely limited in the Skia features it supports (which is why it's so short). It only supports simple color fills, or an extremely limited set of shader types (at this point, just trivial image draws). Given the combinatorial complexity of Skia's API surface, the model of having the blitter know about all the different combinations (color types, effects, etc...) went out the window years ago. Other features that are no longer supported in that model include fancier shaders, color filters, blenders, etc. (You can see many of the decisions that avoid the "legacy" blitters in SkBlitter::UseLegacyBlitter).

The newer blitter system(s) try to decompose the problem -- SkRasterPipelineBlitter being the best example -- it knows how to construct miniature SkRasterPipeline "programs" for any SkPaint, including support for all of the possible effects that might be present. Doing this includes a large amount of code in the blitter and SkRasterPipeline, but also on every SkShader and SkColorFilter in Skia (the onAppendStages virtual functions).

So yes - supporting custom color types would probably mean extending the SkColorType enum, although rendering to planar formats gets even more complicated, because then you do need some higher level knowledge of the memory layout -- RP has no facilities for that. It might be simpler to just render to a supported format and then do a single conversion pass?

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CAOHsvE5e3r%2BZfnRYCck%3DnENM6jCMdG4FJZDWxrp9SWeXOObTgg%40mail.gmail.com.

[D Marc]

Hi Brian,

Thank you again for your answer, that's very kind of you :-)

I was afraid that you would tell me that SkARGB32_Blitter is a limited
legacy implementation. Hmm, although it is limited, do you think it
could still be useful to offer some kind of blitter access at the
public API?

From my preliminary understanding of the implementation, it seems like
it is good enough to support "simpler" painting models like HTML5
canvas, postscript, Qt's QPainter or gdiplus. Would SkARGB32_Blitter
work for gradients?

It might be programmatically simpler to render to a supported format
and then copy-convert to the actual output format, but in some use
cases that may actually involve two copies, one to get the current
pixels and convert them to a supported format to have Skia render onto
that, and then convert it back to the output format.

I am curious, how could your idea of allowing custom ColorTypes be
implemented even if only packed pixel formats were supported and we
ignore planar images for now?

Side question: If I wanted to use Skia to draw to a framebuffer, what
is the recommended way to let SkSurface know that each line contains
some padding at the end?

Thank you for your help!

best regards,
Marc

On Fri, 14 Apr 2023 at 22:29, 'Brian Osman' via skia-discuss

> To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CAMcBjo49TeW5Ss7kNKiaDT30HskbEa5pWhFkyZx1E-JEVhBfPQ%40mail.gmail.com.

[K. Moon]

Re: your last question about padding, this is easily done by specifying a value for rowBytes larger than the minimum required size.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CAOHsvE6uWHa44w6Nd%3DWBPv%2BGRuODQjQqAv7MRNVM8Rv-cdACyA%40mail.gmail.com.

[Brian Osman]

Row bytes is the right answer to deal with padding. Alas, even gradients are no longer supported in the legacy blitters (support for linear gradients in legacy blitters was removed in https://skia-review.googlesource.com/c/skia/+/602496). Note that even when the legacy blitters did support shaders (like gradients), they were still very restrictive: each effect had to implement them explicitly using a "shade this span of pixels" API, and the whole architecture operated on arrays of 4-float RGBA pixels. There's really nothing in Skia that's well suited to rendering directly to planar formats, unfortunately. As it stands now, I'm actively removing even more code related to the legacy blitters (for reasons of code size, maintenance burden, and the fact that they don't really scale as we add new features).

I haven't thought through the custom color types that thoroughly ... there are quite a few spots in Skia that need to know what to do with some data, based on the SkColorType (eg, various switches). Still, I could imagine registering custom color types with a small amount of metadata or callback functions that handle encoding/decoding, as well as letting us know the size.

Example CLs for adding new color types (to get a sense of the problem scale): https://skia-review.googlesource.com/c/skia/+/645637, https://skia-review.googlesource.com/c/skia/+/438219

Note that some of the code in those is related to image codecs (which may not be important for your use-case). Also, anything related to SkVM will soon be irrelevant, as that system is being removed. Unfortunately, the most important thing (if you just want CPU rendering) is the raster-pipeline support, and that code is both tricky and under active development. You could look at adding your own color types in your fork, but I can't guarantee it won't break.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CACwGi-6Vnvo_5cocFdbNEQuX%3D9XX%2BtDo4Je27FZODRBWt4%2BBgg%40mail.gmail.com.

[K. Moon]

On the PDFium team, we're currently in the process of optimizing the performance of the new Skia-based backend. One thing I keep running into is that the low-level code that touches pixel memory is very performance-sensitive: Because of the tight loops involved, it's easy to slow things down a lot by just introducing an extra instruction here or there. We've gotten big speed-ups from making sure we maximize the extent we go through Skia's optimized code paths.

A feature that might be useful for PDFium would be native CMYK or spot color support, but those would all involve channel counts greater than the current maximum of 3 color + 1 alpha, so I'm not sure if that falls under "custom color types" here. I suppose one way of handling them (if the drawing operations are channel-independent) would be to repeat the drawing operations multiple times on the per-channel separation. (This would also work for planar formats, I'd think.) This would involve some wasted work, but depending on the drawing operation, perhaps not that much. This doesn't work if there are dependencies between channels, of course.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CAMcBjo7qxK9hZfwhdJarFo469J6Vx254Ku2ZC1xk%2B_s0nF380g%40mail.gmail.com.

[Nigel Tao]

On Mon, Apr 17, 2023 at 11:17 PM 'Brian Osman' via skia-discuss
<skia-d...@googlegroups.com> wrote:
> anything related to SkVM will soon be irrelevant, as that system is being removed. Unfortunately, the most important thing (if you just want CPU rendering) is the raster-pipeline support, and that code is both tricky and under active development.

SkVM started in 2019 [1] and is deprecated [2]. The raster-pipeline
pre-dates that (e.g. src/core/SkRasterPipeline.h says "Copyright
2016").

[1] https://skia-review.googlesource.com/c/skia/+/216665
[2] https://skia-review.googlesource.com/c/skia/+/670336

Out of curiosity... are there any learnable lessons (or just
interesting anecdotes) from the SkVM experiment? Was register-based
versus stack-based important, in hindsight? Were the corner cases too
hard? Did it not hit the performance originally hoped for? Or has the
existing raster-pipeline simply been optimized in the last few years
to close the gap (and so SkVM wasn't worth the additional complexity
or e.g. security concerns re runtime code-gen)? Is 2023
raster-pipeline significantly different from 2019 raster-pipeline? (I
don't know, I haven't been following Skia that closely; I'm just
curious).

Also, correct my mental model if I'm wrong, but the mapping is:
- raster-pipeline means CPU.
- graphite/ganesh means GPU (with various backends, e.g. Metal,
Vulkan). graphite is the newer thing, ganesh is the older thing.

[John Stiles]

Thanks for the thoughtful questions!

Yes, we have an internal doc which goes over a lot of the reasons. I can look into making it more broadly available. I can summarize some of the bullet points here:

- SkVM was designed first and foremost to JIT shaders into native code. In practice, many of our downstream clients (Chrome, iOS) discourage or ban JIT'ing outright, and so we were not actually getting the fast-path in SkVM for most clients. 

- The JIT had not been security-hardened, and we had already experienced a few cases where it would silently generate incorrect code. There wasn't much motivation to improve it, since our biggest clients didn't want it anyway.

- SkVM had no support for looping or branching, which limited us to ES2-level shaders. When we investigated adding support, we hit several roadblocks. It wasn't insurmountable, but it would involve significant compromises—either worse performance, or much more complexity. So we were basically looking at significant rewrites either way.

- SkVM had a very high setup cost for each paint. If you changed any part of the paint configuration, it would rebuild the entire shader from scratch. Even when the configuration was a perfect match for a previous paint, the cost of finding a match in the shader cache was surprisingly high.

The initial goal was for SkVM to replace Raster Pipeline entirely. However, given the above constraints, we had painted ourselves into a corner—getting rid of Raster Pipeline would be a big perf regression for major clients, but getting rid of SkVM would break Runtime Effects on CPU. We broke the cycle by reimplementing Runtime Effects in Raster Pipeline—I'm working on this now, and it's close to complete. Once this is deployed, it'll be an easy decision to remove SkVM; it's effectively redundant.

I think there are a handful of lessons, to be sure. I wasn't around for the initial planning of SkVM, so I can't fully characterize the decision making process that went into it. To me, at any rate, it seems like they put a lot of eggs into the JIT basket. This seems high-risk move. But, maybe this was a calculated choice—if SkVM were really a quantum leap in performance, the risk might have paid off somehow. Ideally, you'd hedge against that risk somehow, but in practice, SkVM without a JIT really had the deck stacked against it. Another questionable choice was the lack of flow control—if the SkVM design could have supported branches, we might have continued using it for Runtime Effects, even if the performance was bad. But we found that retrofitting flow control after the fact was extremely difficult, and this gave us further motivation to focus on Raster Pipeline instead.

[John Stiles]

Sorry for the very-late followup! I've made the SkSL on Raster Pipeline documentation publicly accessible. Many of the linked docs are still private, but I've replaced internal links with public ones where possible. Hopefully this is a good community resource for others who want to brave the internals of Raster Pipeline.

https://docs.google.com/document/d/1GCQeAGVGHubOCbmULVdXUkNiXdw9J4umai_M5X3JGS4/edit?usp=sharing