Low level graphics

[Diego Sánchez]

Hi,

  Even if this group is about a GUI Lite for C++, I believe this is the best place to present an idea about having a low level infrastructure for dealing with images.

I work in the video industry, building video encoders, decoders, monitors, ... so my work mainly deals with pure pixel crunching (I also do 3D renderings in my spare time). This means that I'm strongly biased towards low level access to individual bits and bytes. However, I believe this would benefit a lot of people out there:

 What I'd like to propose is a class that is conceptually similar to std::string, but geared towards 2D images; and that plays nicely with the standard library; particularly containers and algorithms. I wouldn't call it image, maybe surface would be a better name:

• Template container, defaulting to the almost universal 0xRRGGBBAA pixel format.
• Would treat pixels as arrays of components ( the default pixel format shoud be uint8_t[4] ), using traits to allow for specific formats.
• Provide dynamic allocation of pixels, via a create( size_t width, size_t height, size_t stride = 0 )
• Guarantee that the underlying bits and bytes conform to a sequence of pixels organized in row major order, starting at the upper left corner (providing for line strides).
• Provide iterators on pixels (with automatic skip on the line stride)
• Provide a specific T * data(void) function to access the underlying data. Like std::string::c_str() but not const.
• Allows for block copies of regions, with an optional alpha blend function.
• Allows for channel extraction and recombination.
• Provides a resize function.
• Uses traits to determine if data type is integer (ranges from 0 to std::numeric_limits< Type >::max() ) or floating point (ranges from 0 to 1, allowing for hdr )
• Iteration allows to use local filters via std::transform and similar functions.

I would also add a set of standalone functions to perform colorspace conversions ( RGB 2 HLS and the sort ) on individual pixels.

I'd also like to have things that FFT or PNG and JPG support on the standard library, but I'm not sure when something is considered too complex to be part of the standard library. Note that I choose specifically PNG and JPG because both are covered by ISO standars.

[Tony Van Eerd]

Have you looked at Boost::GIL (Generic Image Library) originally developed by Adobe?

 

 

From: Diego Sánchez [mailto:dsd...@gmail.com]
Sent: Thursday, December 05, 2013 3:00 PM
To: grap...@isocpp.org
Subject: [graphics] Low level graphics

 

Hi,

 

  Even if this group is about a GUI Lite for C++, I believe this is the best place to present an idea about having a low level infrastructure for dealing with images.

.

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

[Diego Sánchez]

Well, GIL is a much bigger beast. I've never used it, and only looked at it a long time ago. I'm under the impression that it's complexity may scare away less experienced programmers, but should look at it again.  Would something of that scope be included in the standards library if there is some interest on it?

I was thinking of something lightweight, to provide the very basic bones that someone would need for a simple graphics task.

[Alex B]

Instead of a single class for a 2D pixel array, I would rather see:

- A color class to represent single color values (make it generic enough to support various formats)

- A generic 2-dimensional array class (which could be used for other purposes than graphics)

 

So an image could be represented as a 2D array of color values.

[Tony Van Eerd]

I find it is best to separate a Color from a Pixel.

 

A Pixel has a particular format for a particular device/hardware/platform/etc.

 

A Color is a color is a colour.  A higher level concept independent of low level pixel stuff.  And a Color can pass between pixel formats.

 

Tony

 

 

From: Alex B [mailto:deva...@gmail.com]

Sent: Thursday, December 05, 2013 4:26 PM
To: grap...@isocpp.org

--
You received this message because you are subscribed to the Google Groups "Graphics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graphics+u...@isocpp.org.
To post to this group, send email to grap...@isocpp.org.
Visit this group at http://groups.google.com/a/isocpp.org/group/graphics/.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/graphics/e6aea83a-8684-48bf-877e-d0fefc0396d3%40isocpp.org.

[Diego Sánchez]

At first I also thought of that approach. However, representing a color as an array of values makes easier the implementation of generic functions, and allows for operations to be applied on different channels regardless of colorspace. This has de disadvantage, however, that you may incorrectly apply an algorith (say you linearly interpolate on HLS colors ).

A generic 2-d array would have to be, well, generic. I believe we need something more specifically geared towards low level image processing. I would expect to have a very specific definition of the ordering of bytes, pixels lines and all that.

[Alex B]

I don't understand the difference you make between colors and pixels. What does a pixel contains appart from a color?

In 3D graphics, we can think about depth values and such. But when you talk about "pixel formats", I don't see why it wouldn't apply to colors as well.

--
You received this message because you are subscribed to a topic in the Google Groups "Graphics" group.
To unsubscribe from this topic, visit https://groups.google.com/a/isocpp.org/d/topic/graphics/3-F4r_r_9mg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to graphics+u...@isocpp.org.

To post to this group, send email to grap...@isocpp.org.
Visit this group at http://groups.google.com/a/isocpp.org/group/graphics/.

To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/graphics/11B5438424E807459A28BD01C92339BA756238ED%40XMB122CNC.rim.net.

[Klaim - Joël Lamotte]

On Fri, Dec 6, 2013 at 4:01 PM, Alex B <deva...@gmail.com> wrote:

I don't understand the difference you make between colors and pixels. What does a pixel contains appart from a color?

In 3D graphics, we can think about depth values and such. But when you talk about "pixel formats", I don't see why it wouldn't apply to colors as well.

Well, colours are range of values like space coordinates: they can be represented in different ways, which we call pixels.
What I think he meant was that you can have a color, say LapisLazuli,  which have different representations depending on the pixel format, which
imply among other things variations of precision and relationships between color channels (as often pixel formats are actually a set of color channels,
while colors are unique values).

I'm not sure how colors can be represented without using a pixel format though.

[Diego Sánchez]

A pixel usually is composed of a color and an alpha, but you may want to add more channels to hold whatever info is needed for the task at hand.  In 3D graphics is when "pixels" may hold things like Z value, G channels; and texels ( texture pixels ) contain a lot of info about the surface (normal maps, specularity, ... ). So an image may contain much more info than mere colours.

My motivation here is that C++ would benefit from having an standard representation of images that's close to the lower levels. It would provide a common ground for interoperability; and my experience is that it is much easier to interface a library that uses raw values than those that provide "advanced" color classes for individual pixels. Also, C libraries tend to use raw buffers, which could be directly mapped into a container like the one I described.

[Diego Sánchez]

Strictly speaking, what we call "color" in image processing is only an approximation to a real color; but one that's good enough so that it usually does't make a difference. 

[Tony Van Eerd]

A pixel is typically a specific representation of a color.  And it changes per platform or even per graphics card or per application or per problem-at-hand (I might need HDR floating point pixels here in the app, but only need black-and-white pixels there in the app).

 

Yet it is useful that a single app with a multitude of pixel formats, only has one Color type.  Typically the Color type is the highest resolution one.  It can serve as the interop between various formats, and between higher level constructs, like what color pen to draw with - the pen color doesn't need to know the underlying pixel format.

 

Now, in theory, you could have an any-pixel-format Image class with an iterator that dealt with Color values, but in reality the per-pixel conversion kills you.

 

You can get pretty far with templatized (on pixel format) image processing functions, that, in the default, non-specialized case, deals with Color values.   But has specializations for all the important cases.  ie

 

template<typename PixelFormat1, typename PixelFormat2>

composite_onto(PixelBuffer < PixelFormat1> bottomDest, PixelBuffer <PixelFormat2> topSource)

{

               PixelFormat1::pixelPtr dst = ...;

                PixelFormat2::pixelPtr src = ...;

                for each pixel...

                                *dst = composite(*dst, *src);

}

 

Then specialize composite() for important combinations of formats - in particular (RGBA, RGBA).

The non-specialized composite() takes two Color values.  This is the generic case.  But that means some conversion probably takes place.

There are even ways to have multiple levels of conversion - ie RGB -> RGBA -> Color.  So that the RGB format *only* supplies a RGB -> RGBA conversion, and doesn't worry about the Color conversion.  (But you can't then use straight compiler-supplied overload resolution to get fine grained ordering.  I actually forget how we got more custom ordering...)

 

You typically don't specialize at the single-pixel-function level.  It is better to specialize at the 1D or 2D level.  (ie 1D means for each scanline or run.)

But the single-pixel function is the fall back default.

 

So you can end up with a non-templatized Image class that contains a runtime pixelFormat member.  You then switch(pixelFormat) at the image or scanline level, calling the templatized functions.   It is OK to do a switch on pixel-format once per image or even once per line.  Just not every pixel!

 

switch (pixelFormat)

{

case RGBA:

                composite_onto<RGBA, RGBA>(image1->data(), image2->data());

etc

 

Or something like that.  The details are fuzzy, sorry -  I've worked on libraries like this 3 or 4 times now at various companies.  But I've forgotten or blended together too many details.  Overall, the design should work even better with C++11 I think.

 

 

 

(By the way, someone asked a while back if there were any image experts on the list.  I didn't pipe up because I still don't consider myself an expert, but I've worked with pixels for about 20 years, mostly in film/tv graphics, including ~10 years at Adobe, which lives and dies over pixels.  But I was also doing UI/threading/architecture/everything..., and leaving the very gritty pixel details to the asm guys.  So I still won't call myself an expert...)

 

 

To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/graphics/CALX6VwpiQKRE-N7%3DN8P%2BM-8uQz0ySnFVttbmCN1wvLMVkJjaqQ%40mail.gmail.com.

[Alex B]

On Fri, Dec 6, 2013 at 11:10 AM, Diego Sánchez <dsd...@gmail.com> wrote:

A pixel usually is composed of a color and an alpha, but you may want to add more channels to hold whatever info is needed for the task at hand.  In 3D graphics is when "pixels" may hold things like Z value, G channels; and texels ( texture pixels ) contain a lot of info about the surface (normal maps, specularity, ... ). So an image may contain much more info than mere colours.

Thanks for the explanation. Since you said that the default format should be 0xRRGGBBAA, I thought that you only wanted to keep color values but now I understand that what you want to keep in your "image"/"surface" data structure can be something else.

So we can be talking about:

- color values

- color formats

- pixel values  (which could contain one or several color values among other things)

- pixel formats

(... or could a color be considered a specialization of pixel?)

 

My motivation here is that C++ would benefit from having an standard representation of images that's close to the lower levels. It would provide a common ground for interoperability; and my experience is that it is much easier to interface a library that uses raw values than those that provide "advanced" color classes for individual pixels. Also, C libraries tend to use raw buffers, which could be directly mapped into a container like the one I described.

Agreed that the data structure should provide an interface to get a pointer to the underlying data (like char* or float*).

However, it should also provide access to strongly typed data (say a generic "color" class and a generic "pixel" class).

[inkwizyt...@gmail.com]

Palette index are another way to store colors, but its now rarely used.

[jason zink]

This is a great discussion, and I think we also touched on the concept of colors, pixels, and their various potential formats and resolutions in another thread as well.  In general, the intention of the library is that we want to let the new users get into the action quickly, and a secondary priority is to keep the 'ceiling' for experts as high as possible.  I think the discussion has actually ranged into both of those areas, so the topic is quite interesting and relevant (at least in my opinion!).

 

With that said, the only real way to get a desired functionality/design into a standard is with a proposal paper, and preferably with a sample implementation.  Our current activities from the most recent SG13 meeting is to wrap Cairo in a basic C++ wrapper as our starting point, and then base the initial proposal on this basic Cairo design.  So if you want to get a head start on a proposal, then please consider how these image concepts will play in that context.

 

So again, just to reiterate - I really like the discussion that is happening here, and it is important to flesh out the design with as many inputs as possible, but in the end you will need to get a proposal paper written up in order to get your concept into the overall standard.  It sounds like Diego probably has significant experience with these concepts, so if you have some sample code for people to try out, that would be a great starting point as well.

 

>> Jason

--

You received this message because you are subscribed to the Google Groups "Graphics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graphics+u...@isocpp.org.
To post to this group, send email to grap...@isocpp.org.
Visit this group at http://groups.google.com/a/isocpp.org/group/graphics/.
To view this discussion on the web visit

https://groups.google.com/a/isocpp.org/d/msgid/graphics/14b61464-9bf3-4bf6-b39b-455ad505840a%40isocpp.org

.

[Diego Sánchez]

Well, interfacing libraries (including what's being done by this group) is what motivated me in the first place. Almost every library/environment out there understands a buffer in the 0xRRGGBBAA format, with lines starting at the top left corner and providing for line stride. This specifically includes Cairo.

But first I believe we should gather info on what are people's thoughts on the matter, and what would be the best approach. So far we have three proposals:

1. "Array of arrays", model each pixel as an array of channels and provide a wrapper class around the image buffer.

2. Use Boost GIL as an starting point.

3. Model it as an strongly typed colour class and a 2D matrix container.

I believe this could be split in two parts:

1. Data model. Do we use specific data types or generic color planes. In other words: balance flexibility versus type safety.

2. Scope. Do we want a lightweight class that provides the basics or a real graphics library.

My point of view is that what would benefit most people is a simple, flexible class for storing image data with just the basics. Take std::string as an example: for most of us, it provides a simple, reliable way of doing everyday tasks like searching for a word or spliting a key/value pair. However, if you where to write an HTML parser you would probably use your internal data structures for all the hard work, and simply provide an option to input data from a std::string. On top of that, you can use c_str() to interface c libraries and it works fine.

So as long as the data representation can be guaranteed to be able to conform with a defined standard I believe it could eventually become a common point for interoperability.

On looking to confirm what would be the best common ground, I've done a quick search on the websites of OpenGL, Cairo, Qt and Microsoft Windows and was surprised to find out that the packed pixel formats (RGB 565 and the like) are still widely supported. If memory serves well, last time I used one of those was some ten years ago on a Windows CE device. Can anyone provide examples of it's use nowadays?

By the way, I also don't consider myself an expert in all that's computer graphics; experienced would be more like it, I've had more than my fair share of pixels over the years.

[Tony Van Eerd]

Are all platforms 0xRRGGBBAA now?  Including cell phones?

And doesn't Windows still have bitmaps starting at the bottom instead of the top?

 

From: Diego Sánchez [mailto:dsd...@gmail.com]
Sent: Monday, December 09, 2013 6:53 PM
To: grap...@isocpp.org
Cc: jason zink
Subject: Re: [graphics] Re: Low level graphics

 

 

Well, interfacing libraries (including what's being done by this group) is what motivated me in the first place. Almost every library/environment out there understands a buffer in the 0xRRGGBBAA format, with lines starting at the top left corner and providing for line stride. This specifically includes Cairo.

---------------------------------------------------------------------

[Thiago Macieira]

On quarta-feira, 11 de dezembro de 2013 16:16:49, Tony Van Eerd wrote:
> Are all platforms 0xRRGGBBAA now? Including cell phones?
> And doesn't Windows still have bitmaps starting at the bottom instead of the
> top?

If I'm not mistaken, OpenGL has the coordinate origin on the bottom, but
that's changeable by settings or by multiplying by -1.

Of course, if you multiply by -1 one too many (or too few) times, you get
upside down renderings. I've seen that. It was the funniest bug ever :-)

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center
PGP/GPG: 0x6EF45358; fingerprint:
E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358

[Diego Sánchez]

It's been a long time since I wrote anything for a handheld, but I'm almost certain both Android and iOs have support for it.

As for Windows, you could select bitmap orientation back in the times of GDI; the bottom up thing was just how it defaulted.

[Diego Sánchez]

But that refers to the higher level of abstraction, how 3D coordinates are mapped into the render area.  Our concern here is the low level arrangement of pixels in memory, be it video memory or an offscreen bitmap.

Another not so funny bug is when you flip your stereo frames. That's a headache!!

[cfy...@gmail.com]

What about the way it handled in FFmpeg?
1. Picture as an object comprised of several more or less separate 'planes'.
2. Plane may represent either color plane (like red or luma, planar formats) or several at once (like bgra or crcb, interleaved formats).
3. Each plane of a single picture may have its own properties (dimensions, line stride).
4. Exact structure is described by 'pixel format', e. g. bgra means interleaved four-channel one-plane image, but yuv420p means three-plane planar image of fullsize luma and quarter-size chroma.

Single plane will be abstracted by 'bitmap' entity and more complex 'image' entity will consist of several bitmaps. This approach allows support for multitude of real-life formats while keeping trivial cases reasonably simple. The most basic bitmap image is just a single interleaved plane with line stride equal line width, i. e. continuous memory chunk of rgba pixels.

As for functionality like stereo images or multi-channeled/multi-layered images, it might be better kept at the higher, application level. Any suitable container (both std and custom) may be used to stack separate pictures into higher-order one.

[mi...@knejp.de]

On Tuesday, December 10, 2013 12:52:35 AM UTC+1, Diego Sánchez wrote:

On looking to confirm what would be the best common ground, I've done a quick search on the websites of OpenGL, Cairo, Qt and Microsoft Windows and was surprised to find out that the packed pixel formats (RGB 565 and the like) are still widely supported. If memory serves well, last time I used one of those was some ten years ago on a Windows CE device. Can anyone provide examples of it's use nowadays?

The RGB565 format is still in use on mobile devices, for example with OpenGL ES (Embedded Systems) as it uses only half the memory(+bandwidth) of RGBA8 and has better aligning. Even if using RGB8 you might get unlucky and have it stored as RGBx on the device to make the pixels aligned on 4 bytes (this includes conversions done by the driver when transferring images to the hardware etc.). If you don't need the precision (and memory) of 8 bits, don't use it.

Besides, are we sure it's 0xRRGGBBAA? As far as I'm aware the notion of RGBA (at least in the little endian world) is to have the red value being the lower byte, which in C++ code would translate to 0xAABBGGRR. Some platforms prefer BGRA, which in C++ notation (on little endian) would still be written as 0xAARRGGBB.

Miro

[Klaim - Joël Lamotte]

On Wed, Dec 18, 2013 at 8:42 PM, <mi...@knejp.de> wrote:

Besides, are we sure it's 0xRRGGBBAA? As far as I'm aware the notion of RGBA (at least in the little endian world) is to have the red value being the lower byte, which in C++ code would translate to 0xAABBGGRR. Some platforms prefer BGRA, which in C++ notation (on little endian) would still be written as 0xAARRGGBB.

If the type represents low-level representation of color, with direct access to raw data but also implementation-defined accessors, and with implementation-defined data layout,
then users wanting cross-platform usage will use accessors (that know how the data is organized internally),
users wanting to have platform-specific optimizations could use information provided by the implementation to manipulate the data directly without accessors.
Would it help?

[sean.mid...@gmail.com]

> Guarantee that the underlying bits and bytes conform to a sequence of pixels organized in row major order, starting at the upper left corner (providing for line strides).

Let's not forget that a 2D array is not all that low-level, either.  Think interlaced or tiled image formats, line strides, etc.  Especially if doing certian kinds of image manipulation, a row of pixels is an awful representation in terms of data locality and you're much better served by various forms of tiling.  Typically hardware handles this transparently in the GPU, but if we're thinking CPU-side "low level" image formats it may be somewhat difficult to make a reasonable implementation that actually works for advanced use cases.

It might be better to aim for a high-level abstraction only intended for a common but incomplete set of use cases.  Trying to make a big "pro grade" library that isn't actually usable for advanced needs and ends up being far more complicated than more casual needs require doesn't do any good.

</two-cents>

[dsan...@cires21.com]

El miércoles, 18 de diciembre de 2013 23:45:49 UTC+1, sean.mid...@gmail.com escribió:

> Guarantee that the underlying bits and bytes conform to a sequence of pixels organized in row major order, starting at the upper left corner (providing for line strides).

Let's not forget that a 2D array is not all that low-level, either.

I can't think of anything that's lower level than bits and bytes, and that's the aim of this proposal: To have a container that defines the exact arrangement of bits and bytes in memory, so it can be easily used to interface almost everything in both C and C++.

[dsan...@cires21.com]

I've been busy with other things, so couldn't devote much time to this. Anyhow, my next step will be to analyze the documentation for a set of frameworks, libraries, specifications... and build a table of supported image formats for interfacing (by image format I mean memory arrangements of pixels). My current list includes:

 - OS:  Windows, OSX, Linux (X11), Android and iOS

 - Frameworks: Qt, GTK

 - 3D: OpenGL, DirectX, Renderman

 - Libraries: Cairo, ImageMagick, OpenCV, ffmpeg

If someone can point me to something I'm missing here, please do.

Just to make it clear: I don't forget that this group is working on a high level graphics library; compatibility with what the group is doing is a must.

[mir...@gmail.com]

Why do not simply take the OpenVG API and "make it" C++ friendly?

OpenVG is an open, well know, industrial strength low level 2D vector graphics standard (see http://www.khronos.org/openvg/ )

[Thiago Macieira]

On quinta-feira, 2 de janeiro de 2014 02:14:07, mir...@gmail.com wrote:
> Why do not simply take the OpenVG API and "make it" C++ friendly?
> OpenVG is an open, well know, industrial strength low level 2D vector
> graphics standard (see http://www.khronos.org/openvg/ )

OpenVG has almost no adoption anywhere. I've only known of a few hardware-
backed OpenVG drivers. Almost everything that even provides it are either
emulated by raster software or done over OpenGL or OpenGL ES. In any case, we
should instead look at those backing implementations instead.

[Ville Voutilainen]

On 2 January 2014 13:33, Thiago Macieira <thi...@macieira.org> wrote:
> On quinta-feira, 2 de janeiro de 2014 02:14:07, mir...@gmail.com wrote:
>> Why do not simply take the OpenVG API and "make it" C++ friendly?
>> OpenVG is an open, well know, industrial strength low level 2D vector
>> graphics standard (see http://www.khronos.org/openvg/ )
>
> OpenVG has almost no adoption anywhere. I've only known of a few hardware-
> backed OpenVG drivers. Almost everything that even provides it are either
> emulated by raster software or done over OpenGL or OpenGL ES. In any case, we
> should instead look at those backing implementations instead.

Concurred. Desktop implementations are rare as unicorns, and they are not
necessarily very efficient (as they may be emulated). My
graphics-pipeline-implementation colleagues report that the over-OpenGL(-ES)
implementations aren't usually very stellar either, and hardware vendors more
or less hate having to even consider supporting OpenVG, and will do that
VG-over-GL anyway. Such a pipeline often has efficiency and correctness issues
due to multiple layers of slight impedance mismatch. There are often problems
with multitasking VG applications and GL applications as well, on some hardware.

Based on the face-to-face meeting minutes at
http://isocpp.org/blog/2013/11/n3825-sg13-discussion-minutes-2013-11-21-jason-zink
it seems that using Cairo as a starting point is a much more reasonable approach
than using OpenVG as a starting point.

[TONGARI J]

2014/1/2 <dsan...@cires21.com>

Seems GIL's pixel and image are what you're looking for for 'low level' components.

http://www.boost.org/doc/libs/1_55_0/libs/gil/doc/index.html

I've built my own graphics library with it, template-heavy, of course ;-)

BTW, has anyone ever consider Skia or AGG?

AGG is more low-level, and Skia is at the same level as Cairo I think.