Rotated font rendering with GWT 1.4 ClippedImage

[Ray Cromwell]

I though some people might get a kick out of my latest blog post. I detail a design for how I used 1.4's ClippedImage + a little assistance from an RPC method and Servlet renderer, to draw text using arbitrary coordinate transforms. The performance is great, I might add, after the initial generation (which can even be moved to compile time if all of your strings come from I18N bundles), and the quality looks almost native.

http://timepedia.blogspot.com/2007/06/gwt-canvas-rendering-rotated-text.html

-Ray

[Reinier Zwitserloot]

Cool stuff. I just added timepedia's blog to my RSS stream. I actually
have 'write a thorough canvas bridge for GWT' on my lost of todos, but
you seem to be doing an excellent job at it. Carry on.

By the way, as a sidenote, while explaining my ReflectedImage widget
(see demo @ http://www.zwitserloot.com/tipit-gwtlib/example) to my
roommate who programs for a job as well, I mentioned that there's no
way to pull that off in Firefox 1.0 (no ActiveX controls that can do
it, and no canvas support either). But then the strategy used by
ClippedImage came to mind - you CAN do it, though it's a bit heavy on
the DOM:

To reflect an image and add a gradient-to-transparancy effect to the
reflection, without using IE-only filters or Canvas, you can simply
render each line of the reflection using 1 DOM div with a height of
1px that uses ClippedImage's strategy to render just the right line.
This div can then be made transparent, or if that can't be done, and
if the 'background' is simply a normal color (usually it is), you can
absolute-place another div with the same dimensions and colour as the
background on top, and change the transparency of that.

Probably too much effort just to support a mostly frivolous effect on
very old browsers, but it goes a ways to show the flexibility of the
whole clipped image thing. I didn't actually write it (yet?), though.

> http://timepedia.blogspot.com/2007/06/gwt-canvas-rendering-rotated-te...
>
> -Ray

[Ray Cromwell]

Thanks,
   Eventually, I'll probably rip-and-refactor the Canvas API out of Chronoscope, and try to get together with Joel/GWT team to work on something official. We had a discussion at GDD, and I think we agree on fundamentals, so it shouldn't be too difficult.

 IMHO, it's desirable for any Canvas API to have the following (not exhaustive list) properties/abstraction

1) Immediate Mode (not DOM/scenegraph oriented)
2) Buffering support (begin/endFrame() )
3) Reversible transformations
4) Text!! Text that obeys coordinate transforms +2
5) Fast-clear
6) Layers/Multiple render targets
7) Display-list would be nice (record list of commands, play them back repeatedly)

This list is not really any different than the design considerations that went into OpenGL/DirectX, You can always build retained-mode vector graphics stuff on top of immediate mode.

 Buffering support means an implementation can, if it wants to, delay flushing (drawing) all the canvas commands until a set point. This not only improves rendering performance (Opera's Canvas supports this abstraction directly), it also allows support for non-CANVAS tag implementations (Flash, Silverlight, SVG, VML, etc) since you can often throw away an entire DOM tree and construct another, vs modifying an existing one, as well as sending an entire buffer of drawing commands across the Flash-Javascript bridge is more efficient.

Layers give you compositing and fast clear. For example, in my usage, I can delete all rendered text DIV or clippedimage IMG tags in O(1) time. (layerElem.innerHTML = '') You can also support more efficient painting, by painting in a single layer and letting the browser composite two aligned CANVAS tags on top of one another. This avoids drawImage calls which are slow in some browsers, and also means you don't have to repaint damaged regions underneath.

Display list would allow one to record a list of moveto/lineto/etc commands and state changes, and then replay them. This can improve performance depending on the underlying drawing mechanism (flash, svg, silverlight, etc) as well as avoiding the overhead of a full JSNI call everytime you want to change canvas state, concatenate a coordinate transform, or build up a path.

For example, I could do a bunch of moveTo/lineTo() calls to construct the letters 'GWT',  saved as a display list, and then just by altering the current coordinate transform or stroke/fill colors, paint this 'GWT' path multiple times over the screen more efficiently than invoking the calls multiple times directly.

I'd like something like:

DisplayList dl = canvas.beginList(");
canvas.moveTo(...)
canvas.lineTo(..)
...
canvas.endList();

Then canvas.callList(dl);

Display lists in particular would vastly improve any Flash or SVG implementations, since the Flash side could cache them until endFrame(), effectively compressing the data sent over the flash bridge, likewise, for other retained mode implementations, the DisplayList can be given an ID and reused repeatedly by reference.

I may be crazy, but I want to see a GWT canvas abstraction enable 'driver' implementations that are efficient enough to support casual game programming. :)

-Ray

[Reinier Zwitserloot]

LGTM.

I actually wrote bomberman in canvas (http://www.zwitserloot.com/
bomberman - not GWT, just js + canvas) and as you can see it runs
quite smoothly (space to start game, sometimes you need to reload if
it doesn't appear, I added some very annoying music so you may want to
turn down the volume, and once a player dies, you can't go back - just
reload to start another game).

JS implementations are getting faster, and CPUs are getting better, so
I don't think you're crazy at all!

[Joel Webber]

Ray,

I'm with Reinier here -- you're not even remotely crazy to think this is all possible & desirable. I'm sure you know from our conversation at GDD that we feel the same way about the immediate mode issue, which is IMHO the most important decision. Most other questions are affected by this in one way or another.

The biggest concerns I have at the moment are:

1. Performance

2. Text

3. Browser support

 

First and foremost, it has to be fast enough to be usable. I keep Google Finance in my mind as a baseline use-case, since it requires a good renderer, uses text, and is interactive. Unfortunately, this takes SVG right out of the running, at least as it's implemented on Firefox. I haven't tried out the Opera & Safari versions so much. But given that Firefox has ~15% penetration, its SVG rendering speed is what matters, and it's abysmal.

Also, it's sad that text has to be put on this list, but the fact (as you guys know all to well) is that text rendering is simply not supported on Canvas, which takes it right out of the running (unless a truly general version of your cool text decal trick can be used).

So that leaves us with VML and Flash (maybe Silverlight one day, but that day's some way off as yet). VML might be a useful fallback on IE, but Flash is the only one that runs across all browsers. I know from my own tests that if you use the 'better' flash bridge (SetVariable, which I believe works in 7, and I know works in 8+), it's quite fast. But it doesn't render rotated text unless you include the converted font in the .SWF (aaaaarrrghh!)

Maybe I'm missing some other option, but it seems to me like Flash is the only viable option for a general-purpose library (at least in the near future). It's not perfect, because of the rotated text problem, but better than nothing. Does this jibe with everyone else's experience? Are we too far ahead of the curve in trying to solve this problem?

joel.

[Ray Cromwell]

I too think Flash is the only viable option (well, applets are a close second, but unusable until Sun solves the startup issue), and Flash has the benefit of being widely deployed, cross-browser, cross-platform, and even on some mobile devices.

1) I found it amazing that Flash's rotated font rendering doesn't extend to arbitrary dynamically loadable OpenType/TrueType, but I'm guessing it's an issue with cross platform portability? (e.g. making sure OpenType/TrueType is available everywhere? Making a legacy licensing issue for Macromedia before Adobe acquired them?)

2) Although I agree that Flash is the "implementation", I think Canvas is the "interface". That is, I would desire an API that is not coupled tightly with
Flash's underlying APIs and capabilities ( e.g. MovieClip)

I think if the Canvas interface is designed appropriately, it will enable a high performance default Flash implementation, as well as leaving the future open
for Silverlight, WHATWG Canvas+Text, etc.

My model for this is OpenGL, I like OGL's high level approach of not exposing underlying details like raw surfaces with pointers, instead using handles. DirectX9-10 have achieved parity and beyond, but I dislike DX's 'direct' approach of exposing hardware more directly vs OpenGL's 'opaque handle' approach.

I think we need to design a 2D equivalent of OGL for GWT. A lightweight procedural interface for drawing that is abstract enough to support varying
implementations, but with the right abstractions to permit easy high-performance implementations.

A case in point is the beginFrame/endFrame() abstraction. On implementations where immediate mode is blisteringly fast, it's a no-op. On implementations where it is better to collect a command buffer and then execute it all at once, it permits better performance (Flash, Opera "GameCanvas" LockCanvas(), etc)

I think we are ahead of the curve somewhat, but this is an extremely exciting area for me, so count me in on any project to build a GWT rendering api. :)

(Can we sneak in our own high speed rendering canvas through Google Gears :) ?)

-Ray

[Joel Webber]

On 6/12/07, Ray Cromwell <cromw...@gmail.com> wrote:

I too think Flash is the only viable option (well, applets are a close second, but unusable until Sun solves the startup issue), and Flash has the benefit of being widely deployed, cross-browser, cross-platform, and even on some mobile devices.

1) I found it amazing that Flash's rotated font rendering doesn't extend to arbitrary dynamically loadable OpenType/TrueType, but I'm guessing it's an issue with cross platform portability? (e.g. making sure OpenType/TrueType is available everywhere? Making a legacy licensing issue for Macromedia before Adobe acquired them?)

Weird, eh? I had to second-guess myself many times over when all the text in my rotation test kept disappearing when rotated! I can only assume it's a legacy thing -- with the Flash player being around so long, and platform text rendering having such a long and sordid history (especially under X).

2) Although I agree that Flash is the "implementation", I think Canvas is the "interface". That is, I would desire an API that is not coupled tightly with
Flash's underlying APIs and capabilities ( e.g. MovieClip) 

I think if the Canvas interface is designed appropriately, it will enable a high performance default Flash implementation, as well as leaving the future open
for Silverlight, WHATWG Canvas+Text, etc.

Yes, a thousand times yes. 

My model for this is OpenGL, I like OGL's high level approach of not exposing underlying details like raw surfaces with pointers, instead using handles. DirectX9-10 have achieved parity and beyond, but I dislike DX's 'direct' approach of exposing hardware more directly vs OpenGL's 'opaque handle' approach.

I think we need to design a 2D equivalent of OGL for GWT. A lightweight procedural interface for drawing that is abstract enough to support varying
implementations, but with the right abstractions to permit easy high-performance implementations.

I've always felt the same way about OGL. It's amazing to me that DX could end up so complex without much performance benefit. I get the impression that the initial designers of the user-space library were trying to optimize the wrong stuff.

A case in point is the beginFrame/endFrame() abstraction. On implementations where immediate mode is blisteringly fast, it's a no-op. On implementations where it is better to collect a command buffer and then execute it all at once, it permits better performance (Flash, Opera "GameCanvas" LockCanvas(), etc)

Funny how such a small decision can make such a big difference. When I was writing my test Flash implementation, this is exactly where I ended up as well. The underlying implementation just shoves accumulated 'drawing list' strings into the Flash engine. Interestingly, the ActionScript VM appears to be quite a bit faster than the JavaScript VM, so it ended up being significantly faster to do stuff like this in the API:

 

    g.beginFill(0xffcc7f, 255);

    g.path("M,93,68,L,96,72,100,73,106,72,108,66,105,63,100,62");

    g.endFill();

This is more or less like an SVG path, with some extra commas to make parsing faster on the Flash side (string.split() will probably always be faster than running a bunch of interpreted code). I'm not 100% convinced this is the best approach, but it is definitely fast. 

One question about Flash: does anyone know exactly which runtime version actually supports the SetVariable() / watch() js->flash communication method? Supposedly SetVariable() is supported in some pretty old versions, but I've heard mention that it only works in newer ones (and Macromedia's doc isn't really clear on the subject).

I think we are ahead of the curve somewhat, but this is an extremely exciting area for me, so count me in on any project to build a GWT rendering api. :)

(Can we sneak in our own high speed rendering canvas through Google Gears :) ?)

That would be really cool, but I'm not holding my breath quite yet :) 

 

joel.