User experience

[Riccardo Canalicchio]

I think that gnustep has left behind the user experience during this years in favors of portability.

In my opinion, users and developers choose cocoa, firstly for the user experience.

We need a properly backed drawing for smooth animations and maybe an accelerated window compositor (if GNUstep choose the "we are a desktop" roadmap).

What do you think guys about an hardware accelerated drawing?

Cairo is great, but is missing some important feature for modern UI.

Currently is not possible to do blurs or shadows, animation are slow and I think that this influence the overall experience of GNUstep.

I know that is possible to use cairo trough OpenGL but right now is missing all the image filter part..

I have looked at Skia, the graphic library used by google chrome:

https://code.google.com/p/skia/

seems very nice and has already implemented most image filter directly on GPU

my two cents

Riccardo

[David Chisnall]

On 22 Dec 2013, at 17:00, Riccardo Canalicchio <riccardo.c...@gmail.com> wrote:

> I think that gnustep has left behind the user experience during this years in favors of portability.
> In my opinion, users and developers choose cocoa, firstly for the user experience.
> We need a properly backed drawing for smooth animations and maybe an accelerated window compositor (if GNUstep choose the "we are a desktop" roadmap).
>
> What do you think guys about an hardware accelerated drawing?

That depends on what you mean by drawing. Aside from a couple of experimental things, I don't think I've seen anyone using hardware acceleration for line drawing recently, because the extra cost of sending all of the commands across the bus offsets everything else.

On the other hand, hardware-assisted compositing is definitely a sensible thing to do. Opal intends to support this, and Ivan's summer of code project made some progress in this direction last year.

Doing it well requires restructuring the back end somewhat so that every drawing context is rendering to a memory buffer and then the window (or, for UIKit, screen) context will handle the compositing.

> Cairo is great, but is missing some important feature for modern UI.
> Currently is not possible to do blurs or shadows, animation are slow and I think that this influence the overall experience of GNUstep.

Cairo does support these things, and Opal lets us use them.

> I know that is possible to use cairo trough OpenGL but right now is missing all the image filter part..

The image filter part is to do with the compositing layer, rather than the drawing layer. Once you have a rendered view in an OpenGL texture, you just need to run a shader on the texture when you do the compositing. Apple has moved to using OpenCL for these, so it's easier to be interoperable than before...

> I have looked at Skia, the graphic library used by google chrome:
> https://code.google.com/p/skia/
>
> seems very nice and has already implemented most image filter directly on GPU

Skia is nice. It's also likely to be supported by NaCL / PNaCl, which would be a very interesting deployment target.

At the moment, a more pressing concern is finishing up the CoreAnimation stuff. This is really important for battery life on modern machines and for smooth effects. It's also a prerequisite for CoreAnimation, which restructures the entire UI around composited layers exclusively - it's a nice addon for AppKit but absolutely essential for UIKit.

We have an unfinished implementation that uses Opal (which implements CoreGraphics, and is another prerequisite for UIKit), and patches to that would be very welcome. CoreGraphics, Skia, and Cairo all implement the PostScript display model, with slightly different names, so an alternative Opal implementation that uses Skia instead of Cairo would be interesting.

We discussed at Cambridge and more or less agreed on using CoreGraphics / Opal as the interface to the rendering back end. We'd then use Opal on all platforms for drawing (the xlib and art back ends have been deprecated for years and need to die) and have the per-platform back ends just be responsible for creating the drawing contexts (possibly even just for creating OpenGL [ES] contexts) and handling input events.

If you're interested in working on any of this then Fred, Eric, or Ivan can give you more detailed suggestions.

David



--
This email complies with ISO 3103

[Riccardo Canalicchio]

Thanks David for your response,

yes UIKit without Core Animation does not make sense at all...

So Opal accelerate only the compositing? it uses cairo on the cpu and then copy the surface on a texture right?

anyway yes I would like to help!

[Ivan Vučica]

Opal uses Cairo, so it uses whatever acceleration Cairo uses (if any).

GNUstep's Core Animation implementation that I worked on uses OpenGL. It's suboptimal right now, but Core Animation is the right way to approach shadows, blurring et al., as you can use shaders to deal with that.

Accelerating drawing 2D primitives itself is, as far as I understand, abandoned by Apple. I may be wrong. There was an implementation of CG that appeared in 10.5 (I think) that tried to use GL; it was faster in edge cases, but not generally.

Again, I may be wrong, but I firmly believe that you want to use Core Animation if you do animations, compositing, etc., and that it makes no sense to try to accelerate drawing 2D primitives using 3D cards.

If you want to contribute to Core Animation, get in touch with me; things that most sorely need to be worked on are:

- scheduling frames for updates (right now it's a game-style infinite loop which draws every frame as fast as the GPU can handle it)

- marking 'non-content contents' as dirty, and only updating when the contents are dirty; for the lack of better term right now, by 'non-content contents' I mean a 'screenshot' that needs to be taken of the entire hierarchy to support shadows, filters and perspective-transformed layers. 'Content contents' on the other hand would be the CGImage or CGContext assigned to the CALayer

- shadow GLSL shader needs to be properly thought out and implemented, esp. the blurring

- we need a masking GLSL shader

- CATransition -- a CAAnimation-style object going from one 'contents' texture to another -- is completely unsupported right now

- dynamic properties need to be generated -- the ones defined with @interface and not-really-synthesized by @dynamic ; this will clean up the CALayer code 'a bit' (by which I mean, a lot)

- Core Image-type filters are cute, let's throw that in

This is approximately in decreasing order of importance (@dynamic thing is not important at all, but will make code implementing setters and getters a lot shorter).

[Riccardo Canalicchio]

Ok, I can help on the shaders field or the frame scheduling ( I'm not really experienced with xorg) the upstream is on svn? I need to study the current code...
Do you have any paper of the overall design of CoreAnimation?
Currently gnustep seems to have not a roadmap but this project has one?

Just another question are you sure to use directly opengl?

For example Clutter uses CoGL that wraps opengl and make a common api between opengl and opengles; skia also supports both opengl and gles... i have seen a lot of mails talking about uikit but opengl is not supported on mobile...

[David Chisnall]

On 23 Dec 2013, at 16:09, Riccardo Canalicchio <riccardo.c...@gmail.com> wrote:

> For example Clutter uses CoGL that wraps opengl and make a common api between opengl and opengles; skia also supports both opengl and gles... i have seen a lot of mails talking about uikit but opengl is not supported on mobile...

GLES is a subset of (recentish) OpenGL. Sticking to the GLES subset should let things work on the desktop just fine.

David

-- Sent from my brain

[Ivan Vučica]

Hi Riccardo,

On Mon Dec 23 2013 at 4:09:33 PM, Riccardo Canalicchio <riccardo.c...@gmail.com> wrote:

Ok, I can help on the shaders field or the frame scheduling ( I'm not really experienced with xorg) the upstream is on svn? I need to study the current code... 

Yes, all the code you want is on SVN. See the 'quartzcore' directory (we really need to rename this).

Do you have any paper of the overall design of CoreAnimation?

There is no such paper. It took me a while to figure out how best to implement Core Animation while staying compatible; in fact, that is an even more important part of my GSoC2012 work than the actual implementation itself.

The implementation itself is the documentation, and I tried to iron out even the edge cases. Sadly, there are still cases where even the timing implementation that's in our code isn't fully compatible, but you'll notice that only in edge cases. And if someone is depending on Apple-compatible behavior, well, we can think about it when it's reported.

Currently gnustep seems to have not a roadmap but this project has one?

Not really, but it has a list of things you need if you want to make our implementation actually useful.

Just another question are you sure to use directly opengl?

I'm completely sure.

For example Clutter uses CoGL that wraps opengl and make a common api between opengl and opengles; skia also supports both opengl and gles... i have seen a lot of mails talking about uikit but opengl is not supported on mobile...

Clutter is a scene manager, like Core Animation. Let's not include a middleware that doesn't need to be there. That goes for any other API wrappers; Core Animation API itself is a wrapper around OpenGL.

Skia is, to the best of my knowledge, a Cairo and Core Graphics analogue, and deals primarily with rendering of 2D primitives. Core Animation deals with exposing compositing, 3D transforms and filtering to an Objective-C application in a nice API which supports implicit creation of CAAnimation objects. Skia could be used to provide content for CALayers, but there is no need to; we have Opal (our implementation of Core Graphics) for that.

And regarding OpenGL not being supported on mobile, I'd advise you to compare the API of OpenGL and OpenGL ES. You'll see a lot of similarities. In fact, as David said, if you write OpenGL ES code, you need to do only minimum adaptations.

At one point I had OpenGL ES compatibility in GNUstep's implementation of CA; I didn't test it in a long time by the time I wrapped up the GSoC project. In fact, it almost certainly doesn't work, considering that I added (very dumb and slow) shadow support by using shaders, which aren't supported in OpenGL ES 1.x, but are supported in OpenGL ES 2.0. The solution would be to transition to OpenGL ES 2.0, which would require use of (simple) shaders everywhere, as ES 2.0 tried to do away with as much of fixed function pipeline as possible; it's not a big deal, but it would take (only?) a few hours.

On Tue, Dec 24, 2013 at 1:05 AM, Lundberg, Johannes <joha...@brilliantservice.co.jp> wrote:

Hi

As you know, CG and CA is definitely a necessity for our project. As soon as I can get my hands on someone who knows their ways around Open[GL/CL/VG] etc I will put them to work on helping you implementing this in GNUstep :)

--

Johannes Lundberg

I'll be more than happy to provide initial planning and advice, as well as (pending positive resolution of some paperwork) code :-)

OpenGL and OpenGL ES knowledge is probably enough, no need to worry about OpenCL and OpenVG.

[Tom Bruno]

I'm currently looking and interested in the quartzcore project. I've been reading through the source and all the threads that I can find.  I've been able to compile, install & run the demos but have a few questions.  I see your notes about opal requiring your opal-nsfonthacks.patch.  It seems this patch can no longer apply. In addition to that issue, it also seems that when I run all the demo fonts are not visible.  I'm currently using clang-3.4 and a self build of svn trunk for all modules/libs etc.

  Tom

_______________________________________________
Discuss-gnustep mailing list
Discuss...@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnustep

[Ivan Vučica]

The patch is no longer required as Eric has helped by renaming the relevant classes when I was doing this year's GSoC project.

Sadly I don't know about the fonts issue; they should be drawn by Opal / Core Graphics. The relevant thing is done in the demo anyway, and not in QuartzCore code itself, so if you can figure out a way to draw fonts -- a patch will we very welcome :-)

(This is because I cannot work on QC right now, so I can't look into the issue. I will help with my thoughts and advice in any way I reasonably can, though.)

[Tom Bruno]

I see a lot of the empty headers and .m files that I suppose represent the "unfinished" portion.  I'm curious as to how you came about the API/Headers that you would implement.  Some of these headers are only available on iOS platforms (CAEAGLLayer, CADisplayLink).  I read a post from while you were working on the project that said you were going to try and stay away from CoreVideo but things like QuartzCore/CVDisplayLink.h (the Mac Equiv to iOS's CADisplayLink.h) are part of CoreVideo.  The CVDisplayLink is a common way to drive the OpenGL frame drawing on OSX.

I guess what I'm trying to ask is what was your thinking and where were you headed with selecting CA* files that only exist on iOS, or was this just something that happened in passing?  I'm trying to get a feel for where you left off and were going.

 Tom

[Ivan Vučica]

Please look at "authors" section - in a lot of cases I was being very courteous by leaving the previous author in, despite actually filling out the implementation and fixing the headers :-)

So the empty files are more of a remnant that "will be useful some day".

I picked what I would implement based on following:
* we need something useful that animates
* that doesn't include integration with AppKit - larger effort than realistically possible
* there is a non-integrated way to render content that Apple exposes: CARenderer
* content is represented by CALayer
* CALayer needs to support CGImageRef content and dynamically provided content drawn into CGContextRef
* animation is represented by CAAnimation
* simplest tweens are implemented with CABasicAnimation
* timing functions need some awesome math to work correctly, and need to be carefully implemented to work in the way OS X and iOS developers use them (or at least use them most often)
* CALayers support 3D transforms and, in fact, depend on them for doing even simplest rotations - we need matrix math
* shaders were implemented so we can do blurring [unfinished] and masking [not done yet] in hardware
* we need an observer to create implicit animations

End result was the nice (simple, but nice) demo that included implicit and explicit animations, shadows, elementary content clipping, 3D transform with a perspective matrix.

And work on anything else is now a matter of plugging right component in the right place. Need a more complex variety of CAAnimation? Just write it. Hard unfinished parts (such as correctly functioning -nextFrameTime plus a way for CALayer or CAAnimation to notify CARenderer to notify hosting framework or application that there was a change in timing and that -nextFrameTime changed).

If I left anything important out, let me know and I'll be happy to provide reasoning.

What I didn't implement?
* CVDisplayLink depends on having Core Video. Or at least a nice way to sync with, well, vsync. And it needs a place to live. QC, despite its misleading name, is a Core Animation implementation right now; it isn't the right place for it. -- besides, CVDisplayLink would be interesting to the hosting app/framework, not CA.
* Same for CADisplayLink

And why are the headers such as CAEAGLLayer there? They were put there by someone else, but do no harm and might be useful someday. Why not have a CAEAGLLayer on desktop Linux? We have implementations of OpenGL ES on desktop anyway.

Hope this clears stuff up :-)