Long-term plans for the PlayN graphics architecture

232 views
Skip to first unread message

Joel Webber

unread,
Dec 5, 2011, 2:34:00 PM12/5/11
to pl...@googlegroups.com, Michael Bayne, Matthew Mastracci
All,

I've been promising a few significant changes for some time now:
- To replace the dom-based html rendering backend with a canvas-based one,
- replace the java2d rendering code with lwjgl, and
- unify the two (three after making the lwjgl change) opengl rendering backends with a single shared one.

I sent a (as-yet-still buggy) patch for the first of these a few weeks ago (http://code.google.com/p/playn/issues/detail?id=97), but as I was about to go finish the remaining bugs, I began chatting with Matt (cc'd) about it, only to discover that he's relying on the performance of dom-based rendering on the iPad (where it performs *much* better than canvas). But the existence of the dom code makes it fairly nasty to fix http://code.google.com/p/playn/issues/detail?id=96, which in turn makes it hard to clean up some of the weirder parts of the relationship between Image, Canvas, Surface, and Layer.

Now let's add two more goals to the mix:
- The ability to add "direct-paint" layers
  (meaning the kind where you get a callback when it's time to paint, and you're painting directly into the same target surface as the compositor)
- 3D layers (both surface-backed and direct-paint).

The idea of direct-paint layers might require a little context. The problem we have right now, for example in the "cute" sample, is that many types of games want to paint everything (or at least the game itself, if not things like score overlays) from back to front on every frame. Take the example of an isometric world -- forcing the developer to shoehorn all the iso tiles and game objects into separate layers is both extremely awkward and fairly inefficient. So instead you end up painting the entire game area to an offscreen surface, which itself gets blitted to the main surface during compositing. This isn't too terrible on desktop machines, where you typically have very high fill-rate video cards and plenty of memory; but it's pretty inefficient on mobile devices, especially tablets that have slower GPUs, less memory, less bandwidth, and big screens. That extra large render-buffer, and all the extra copying, is really wasteful.

If you consider 3D layers, they have largely the same problem -- you almost *never* end up using any 3D rendering system without repainting the entire 3D surface on every frame. Which means that every 3D game (or even most 2D games that use 3D hardware) will be wasting a bunch of fill-rate and memory on an unnecessary offscreen surface.

So what's the relationship among all these things? Well, there are currently two things keeping me from implementing direct-paint layers -- the Flash platform, and the dom-based renderer in the HTML platform. There's really no sane way I can think of to implement direct-paint layers in the dom renderer (you might suggest implementing it with implicitly-created canvases, but that won't work because you'd need to know the size of the canvas in advance, but one of the most useful properties of direct-painting is that the layer is just part of the transform stack, and painting "out of bounds" is both legal and useful). And while I'd really like to handle iOS devices by targeting them natively, it's going to take a while before we get there, and in the meantime the browser's dom renderer is a reasonable fallback for some games.

As for Flash, I know there's been a lot of contention around whether it's worth keeping around. On one hand, it's kind of an albatross, and those of us responsible haven't really been maintaining it. On the other hand, I've talked to a few people working on commercial games who've said it's fairly important (E.g., one estimated they'd lose ~30% of their existing customers without it). On the third hand (?) no one's stepped up to maintain it either, which is a good way to signal the strength of one's interest in it.

One thing I *do* need in order to reason about the impact of supporting Flash on our future plans is to know how feasible it is to support direct-paint layers efficiently within it. I'm no Flash expert, but my (possibly flawed) understanding is that it's not possible to support this rendering method efficiently. If that's the case (Flash experts, please chime in or I'll just assume I'm right about this!), then I think it's unrealistic to support Flash much longer.

And for anyone (Matt?) relying on dom/css performance as a stopgap for supporting iOS devices, how important do you think this is for your projects? And do you think there is a realistic way to get acceptable performance from direct rendering on this platform (which I presume means "with canvas"), perhaps by reducing the resolution and possibly scaling up with an accelerated transform?

For those of you who've made it this far, thanks for reading to the end. Making sensible decisions about this stuff requires everyone's input, because there's no "right" approach -- all the possibilities involve significant tradeoffs. But I feel it's very important to get this system into a state where the graphics architecture will serve us reasonably far into the future. And that means triangulating against what we expect the state of these platforms to be five years out.

Thanks,
joel.

consiliens

unread,
Dec 5, 2011, 3:31:15 PM12/5/11
to pl...@googlegroups.com
On 12/05/2011 12:34 PM, Joel Webber wrote:
>
> One thing I *do* need in order to reason about the impact of supporting
> Flash on our future plans is to know how feasible it is to support
> direct-paint layers efficiently within it. I'm no Flash expert, but my
> (possibly flawed) understanding is that it's not possible to support
> this rendering method efficiently. If that's the case (Flash experts,
> please chime in or I'll just assume I'm right about this!), then I think
> it's unrealistic to support Flash much longer.

What about supporting just Stage3D (which is now shipping in Flash 11)?
I agree about not supporting the legacy Flash drawing APIs.

Ray Cromwell

unread,
Dec 5, 2011, 7:01:40 PM12/5/11
to pl...@googlegroups.com, Michael Bayne, Matthew Mastracci
Joel, last time this came up (Flash and direct-rendering), I remarked
that drawing in Flash by using Bitmap drawing ops would be less
efficient than using ImageLayer -> Flash Sprite. That's still true,
but it may be moot. Let's not worry about performance too much at this
point and make the changes. Flash performance will get a big kick with
Flash11 assuming I port the GL HTML stuff over to use the Flash 3D
APIs.

What I can do is, after you submit your changes, I can suggest any
neccessary API changes that might allow the flash backend to render
more efficiently.

-Ray

Michael Bayne

unread,
Dec 7, 2011, 1:34:39 PM12/7/11
to Joel Webber, pl...@googlegroups.com, Matthew Mastracci
On Mon, Dec 5, 2011 at 11:34 AM, Joel Webber <j...@google.com> wrote:
> I've been promising a few significant changes for some time now:
> - To replace the dom-based html rendering backend with a canvas-based one,

I'm strongly in favor of this. I want to spend more time helping debug
your new canvas backend, but I got yoinked off onto a wacky side
project. I'll hopefully be able to get back to at least part-time
PlayN development soon.

> - replace the java2d rendering code with lwjgl, and

I don't have strong opinions on this front, other than a mild aversion
because it will make it impossible to deploy PlayN games as unsigned
applets. If it weren't for Minecraft's kajillion dollars, I would
write applets off completely. I suppose Minecraft was a signed applet
that used LWJGL anyway, so maybe I shouldn't care.

> - unify the two (three after making the lwjgl change) opengl rendering
> backends with a single shared one.

I think this is going to be impossible due to the pervasive use of
typed arrays (WebGL), versus NIO array buffers (LWJGL and Android GL).
Maybe we can do some GWT magic here and use NIO buffers everywhere and
have those magically be rewritten as typed arrays during compilation,
but I'm wary of introducing yet another abstraction over buffers.

> Now let's add two more goals to the mix:
> - The ability to add "direct-paint" layers

I'm strongly in favor of this. I have dozens of canvas or surface
layer uses that I am dying to rewrite to render directly.

> - 3D layers (both surface-backed and direct-paint).

This seems like a giant distraction to me, but I don't want to rain on
anyone's parade.

> And for anyone (Matt?) relying on dom/css performance as a stopgap for
> supporting iOS devices, how important do you think this is for your
> projects? And do you think there is a realistic way to get acceptable
> performance from direct rendering on this platform (which I presume means
> "with canvas"), perhaps by reducing the resolution and possibly scaling up
> with an accelerated transform?

I've been doing a lot of testing of my new game on an iPhone 4 with
the DOM backend, and I find it to be painfully sluggish, and my game
is not even remotely graphically intensive. This is in Mobile Safari
as well which uses the JavaScript JIT compiler. If you actually wanted
to ship an app on iOS using the HTML5 backend, you'd have to use
something like PhoneGap and give up the JavaScript JIT and take a
great big performance hit on top of the already crappy performance.

While it would be unfortunate for things to become even more sluggish
by replacing DOM with Canvas, I don't think it's viable to ship a game
on iOS using HTML5. Either we come up with a native compilation
approach for iOS or we don't support it. Things might be decent on an
iPhone 4S or iPad 2, but it's a pretty sorry state of affairs if we
can't do super simple 2D graphics on anything except the fastest
multi-core iOS devices.

I'm strongly in favor of switching to a simpler backend where the core
requirement is the ability to do accelerated drawing/blitting into a
frame buffer. That's the right architecture for future development and
it will give us a good balance of simplicity of the backend
implementations and room to maneuver so that we can achieve good
performance across all the supported platforms.

-- m...@samskivert.com

Mike Fleischauer

unread,
Dec 7, 2011, 6:53:14 PM12/7/11
to PlayN

> As for Flash, I know there's been a lot of contention around whether it's
> worth keeping around. On one hand, it's kind of an albatross, and those of
> us responsible haven't really been maintaining it. On the other hand, I've
> talked to a few people working on commercial games who've said it's fairly
> important (E.g., one estimated they'd lose ~30% of their existing customers
> without it). On the third hand (?) no one's stepped up to maintain it
> either, which is a good way to signal the strength of one's interest in it.

First off, I don't really care if this feature is kept or not, frankly
it sounds more like smoke and mirrors than a feature at this point, so
I'm fine with axing it.


However, I just want to clarify one thing, the skill-set difference
between consuming a library that utilizes Flash and writing a Flash
cross compiler are massively different! I wouldn't look towards the
lack of volunteers as indicating a lack of interest. Frankly, outside
of the Google-sphere, the number of people with experience writing GWT
back-ends is like... 6. That's one of the major problems with PlayN
from the whole expecting external developers to chip in part, first
off, the Java ecosystem ( Ant... Maven... Eclipse... on it goes ) has
an exorbitant mental cost of entry to start with, but then on top of
that, you are built fairly heavily on proprietary ( open, but
proprietary ) Google tech to start with. Unlike most other open source
projects, there simply aren't too many developers, even experienced
Java developers, that can easily jump in and help.

TL;DR, a Flash back in is either going to be a Google driven feature,
and Adobe driven feature, or something you should drop completely.

Matt Mastracci

unread,
Dec 7, 2011, 9:48:08 PM12/7/11
to Joel Webber, pl...@googlegroups.com, Michael Bayne
I'd prefer not to drop the DOM renderer. I've got a full-fledged game, with good performance, up-and-running on the iPad 1 using the DOM engine. The same game also works well on the iPhone 4. Getting here was no easy feat, however: it required careful timing when adding new ImageLayers to the screen, lots of ImageLayer pooling, and spacing out canvas operations (such as font drawing) to avoid dropping the frame rate. 

I also experimented with canvas-based rendering performance using Joel's XNA platformer port. I wasn't able to get performance on the iPad 1 up past 2-5 fps, and that's with only a handful of tiles on the screen. The iPad 1 is going to be around as a potential platform target for at least another year and a half before it hits Apple's effective end-of-life. There's still a *lot* of unexplored space around optimization of canvas on iPad 1, but it didn't appear too promising.

This suggestion might be contentious, but I think that PlayN adopt the concept of platform 'profiles'. For 'low-end' platforms (DOM and Flash), we could offer today's set of image/canvas/group layers as the baseline profile. These platforms wouldn't get support for the new direct-rendering surfaces that would appear in the 1-up platform profile. The proposed 3D layers in the future would be yet another profile.

I'm also -1 on dropping Flash, even though it's not had a lot of life in it. I think that we'll eventually be able to bring it back from the dead, at least on the level of the basic profile I mentioned above. I definitely don't think we need to worry about supporting direct painting and 3D in the Flash engine. 

Also, since the entire set of low-end devices that 1) don't support flash, and 2) have a marketshare worth worrying about (IMHO of course) are Webkit-based, I would suggest that we could rip out support for all non-Webkit browsers in the DOM renderer after the Canvas renderer lands. 

We should probably decide on the next steps for the CanvasLayer/CanvasImage API soon. I've got some code in place that allows CanvasImage to work properly inside of an ImageLayer, at least on Webkit. If we add the concept of 'flushing' canvases to update the associated layers, I think we'll be able to shoehorn support into the Flash engine at some later date. It would be nice to have this behaviour spec'd out one way or another. 

Matt Mastracci
Twitter@mmastrac
Bloggrack.com

Ray Cromwell

unread,
Dec 7, 2011, 11:19:06 PM12/7/11
to pl...@googlegroups.com, Joel Webber, Michael Bayne
I'll try to resurrect Flash over the holiday. The biggest reason it's
broke is because GWT2.4 changed internal compiler classes I was
depending on, the fix requires me to hack the compiler source to add
back the hooks I need, which I will do for the GWT 2.5 release. In the
meantime, if people are willing to build off of a trunk release
snapshot, I could probably fix it quicker.

consiliens

unread,
Dec 8, 2011, 12:51:35 AM12/8/11
to pl...@googlegroups.com
On 12/07/2011 09:19 PM, Ray Cromwell wrote:
> I'll try to resurrect Flash over the holiday. The biggest reason it's
> broke is because GWT2.4 changed internal compiler classes I was
> depending on, the fix requires me to hack the compiler source to add
> back the hooks I need, which I will do for the GWT 2.5 release. In the
> meantime, if people are willing to build off of a trunk release
> snapshot, I could probably fix it quicker.

Does the resurrection include improving the AS3 output as detailed in
GwtFlashTodo?

Simen S.

unread,
Dec 8, 2011, 8:00:49 AM12/8/11
to PlayN
On Dec 8, 5:19 am, Ray Cromwell <cromwell...@google.com> wrote:> I'll

try to resurrect Flash over the holiday. The biggest reason it's>
broke is because GWT2.4 changed internal compiler classes I was>
depending on, the fix requires me to hack the compiler source to add>
back the hooks I need, which I will do for the GWT 2.5 release. In
the> meantime, if people are willing to build off of a trunk release>
snapshot, I could probably fix it quicker.

Sounds awesome. I understand that the flash backend feels like a low
priority for many here, as I assume your focus is on the html5 target,
hence flash feels a bit redundant. But for us, who make social games
with facebook and google plus as our main market, it is an important
target. Rough estimates based on our google analytics tracking on our
other games indicate that around 30% of our users right now do not
have support for canvas. Giving up 30% of our users would be a
crucially big hit to us.

I fully understand though that maintaining something like that is up
to us the users, and I would love to do it, but right now we are still
in the game logic phase, but down the line I am sure we can try to
contribute. But just as Mike points out, the initial investment to get
a grasp of how it all works is huge, and it is not my area of
expertise, but it by no means we have no interest in it. And I would
be more than happy to work off trunk of gwt, if that is what is needed
to have a working flash target, not a unreasonable requirement imo.

Would also just like to thank everyone that works hard on playn and
has gotten it to where it is now, and I will for sure contribute more
as we go more into the polish and bug fix part of our project!

/Simen

Joel Webber

unread,
Dec 12, 2011, 5:09:45 PM12/12/11
to Michael Bayne, pl...@googlegroups.com, Matthew Mastracci
[Just a quick followup to address some comments on issues related to 3D and GL, separate from the other thread]

Le 7 décembre 2011 13:34, Michael Bayne <m...@samskivert.com> a écrit :
On Mon, Dec 5, 2011 at 11:34 AM, Joel Webber <j...@google.com> wrote:
> - replace the java2d rendering code with lwjgl, and

I don't have strong opinions on this front, other than a mild aversion
because it will make it impossible to deploy PlayN games as unsigned
applets. If it weren't for Minecraft's kajillion dollars, I would
write applets off completely. I suppose Minecraft was a signed applet
that used LWJGL anyway, so maybe I shouldn't care.

Yup, Minecraft seems to get by with an applet, and it definitely uses LWJGL. Though it would be a whole lot cooler if it could run directly on the web :)

That said, I think that using LWJGL in the Java implementation actually makes it more feasible to ship applets/java-desktop games, because the performance is likely to be orders-of-magnitude better than Java2D, and with higher-quality rendering (especially scaled filtering) to boot.

> - unify the two (three after making the lwjgl change) opengl rendering
> backends with a single shared one.

I think this is going to be impossible due to the pervasive use of
typed arrays (WebGL), versus NIO array buffers (LWJGL and Android GL).
Maybe we can do some GWT magic here and use NIO buffers everywhere and
have those magically be rewritten as typed arrays during compilation,
but I'm wary of introducing yet another abstraction over buffers.

Stefan Haustein (who did the majority of the GL work on the old gwt-quake port last year) actually already got pretty much everything in nio working on top of typed arrays (made the port a hell of a lot easier). We'd have to be careful to make sure this stuff's getting optimized sensibly, but I'm pretty sure we would end up with most of the methods (especially get/set) being inlined out. There would likely be an unavoidable one-object-wrapper overhead for each buffer, but most actual calls would just turn into buffer._impl.method(...) at the call-site.

Again, the main motivation here is to share the GL code, which I think still needs a lot of work. There's a *lot* of room for optimizing vertex/element-buffer usage, we need to manage texture memory more actively, and so forth. But it would be really nice if we could write this code once, rather than having to duplicate it all over the place as we're doing now.

> - 3D layers (both surface-backed and direct-paint).

This seems like a giant distraction to me, but I don't want to rain on
anyone's parade.

I probably didn't make this clear enough, but I'm bringing this up primarily in the context of "clearing the way for the future", not because I think we should dive into it right now. As I've mentioned elsewhere, it's not even clear to me that "raw GL" is the right abstraction, and it will probably take a while before we can know how "3d on the web" is going to shake out (e.g., Microsoft might decide to go totally off-the-rails on this).

Anyway, all of this can wait until such time as we sort out the 2D API stuff, which is what I'm going to concentrate on for the time being.

Cheers,
joel.

Michael Bayne

unread,
Dec 12, 2011, 5:39:40 PM12/12/11
to Joel Webber, pl...@googlegroups.com, Matthew Mastracci
On Mon, Dec 12, 2011 at 2:09 PM, Joel Webber <j...@google.com> wrote:
> Stefan Haustein (who did the majority of the GL work on the old gwt-quake
> port last year) actually already got pretty much everything in nio working
> on top of typed arrays (made the port a hell of a lot easier).

Ooh, awesome! I'd love to see that make its way into
user/super/com/google/gwt/emul/java/nio.

This is especially poignant, given that I recently spent six hours
debugging a problem that came down to the fact that in
src/com/google/gwt/typedarrays/client/Uint8Array.java and friends, the
subarray methods:

public final native Uint8Array subarray(int offset, int length) /*-{
return this.subarray(offset, length);
}-*/;

are wrong.

The subarray methods take start+end, not offset+length. Of course,
once things are compiled into JavaScript, you get zero useful feedback
when all of your code has gone horribly wrong because your typed
arrays are truncated to zero length arrays (or weird length arrays
when length happens to be greater than offset), and JavaScript returns
NaN instead of doing something useful (like throwing an exception)
when you ask for an array element that's out of bounds. Yay for sloppy
languages, they're so user friendly.

-- m...@samskivert.com

Joel Webber

unread,
Dec 14, 2011, 4:35:23 PM12/14/11
to Michael Bayne, pl...@googlegroups.com, Matthew Mastracci
Yikes! The nio stuff in there was forked from the old gwt-quake code ages ago, and in many cases the interfaces have actually evolved over time (yeah, yay for sloppy languages -- at least you don't find out until the world collapses around your ears, long after the offending call is off the stack...).

Stefan's done a lot of cleanup work on the quake code since then, so it should be in a much better state. It will probably be a while before I can get around to this, as I need to sort out all that Canvas[ImageLayer] stuff first, but once it's in a reasonable state (and we can finally deprecate CanvasLayer), this will be next on my list.

Johannes

unread,
Dec 14, 2011, 7:17:31 PM12/14/11
to PlayN
On 14 Dez., 22:35, Joel Webber <j...@google.com> wrote:
> > On Mon, Dec 12, 2011 at 2:09 PM, Joel Webber <j...@google.com> wrote:
> > > Stefan Haustein (who did the majority of the GL work on the old gwt-quake
> > > port last year) actually already got pretty much everything in nio
> > working
> > > on top of typed arrays (made the port a hell of a lot easier).

> Yikes! The nio stuff in there was forked from the old gwt-quake code ages


> ago, and in many cases the interfaces have actually evolved over time
> (yeah, yay for sloppy languages -- at least you don't find out until the
> world collapses around your ears, long after the offending call is off the
> stack...).
>
> Stefan's done a lot of cleanup work on the quake code since then, so it
> should be in a much better state. It will probably be a while before I can
> get around to this, as I need to sort out all that Canvas[ImageLayer] stuff
> first, but once it's in a reasonable state (and we can finally deprecate
> CanvasLayer), this will be next on my list.

I'm playing around with this stuff at the moment. I looked briefly in
the gwt-quake source code and I'm wondering if it's not expensive to
create a new typed WebGl array out of a nio array in every draw call?
As I see it at the moment the reason for emulating nio was to make the
port to WebGl much easier. But is it needed in a fresh enviroment?

Instead of emulating java.nio I created a generic 'Buffer' interface
which is implemented in the target platform. Like PlayN does it for
many other things.

Joel Webber

unread,
Dec 16, 2011, 11:15:59 AM12/16/11
to pl...@googlegroups.com, Michael Bayne
That may well be the right approach. nio is a bit on the "full-featured" side (read: bloody complicated), so it might be worthwhile to abstract it out under a simpler "closer-to-js-typed-arrays" API. I suppose that would mean we'd have to wrap the nio types, but that should be pretty efficient on the JVM.

The open question is how much existing Java GL code (that's actually worth porting) already uses nio, and could be ported fairly quickly. It definitely made porting Quake II a lot easier to shim nio, because the existing Java port used it heavily. I don't know of much else that I'd want to port, but Michael might have some opinion on this -- how much easier would it be, e.g., to port Spiral Knights to PlayN if its eventual 3D support used the nio interfaces?

Michael Bayne

unread,
Dec 16, 2011, 12:45:17 PM12/16/11
to Joel Webber, pl...@googlegroups.com
On Fri, Dec 16, 2011 at 8:15 AM, Joel Webber <j...@google.com> wrote:
> Michael might have some opinion on this -- how much easier would it be,
> e.g., to port Spiral Knights to PlayN if its eventual 3D support used the
> nio interfaces?

The 3D library that we wrote for SK
(http://github.com/threerings/clyde) uses NIO in 42 source files (out
of 728), with the following breakdown:

23 import java.nio.FloatBuffer;
19 import java.nio.ByteBuffer;
14 import java.nio.IntBuffer;
11 import java.nio.ShortBuffer;
6 import java.nio.DoubleBuffer;
4 import java.nio.LongBuffer;
4 import java.nio.CharBuffer;
2 import java.nio.ByteOrder;
1 import java.nio.charset.Charset;
1 import java.nio.channels.FileChannel;

There's essentially no use of NIO in the main game code (modulo some
Steam integration that uses a combination of NIO and native code).

So it's probably not that big a deal. It would probably take a couple
of hours to change all the uses of NIO directly over to TypedArrays or
some other TypedArray-like buffer abstraction (maybe even less with
some judicious use of Perl).

That said, something like SK is so complex that we would probably not
want the intermediating layer of PlayN and would instead just port
Clyde directly to Android or GWT+WebGL. Experimenting with the former
is on my long list of things to do. Though after seeing Chrome's
performance lately, maybe I should also experiment with the latter. :)

-- m...@samskivert.com

Michael Bayne

unread,
Dec 19, 2011, 1:57:10 PM12/19/11
to Joel Webber, pl...@googlegroups.com, Matthew Mastracci
On Mon, Dec 5, 2011 at 11:34 AM, Joel Webber <j...@google.com> wrote:
> I sent a (as-yet-still buggy) patch for the first of these a few weeks ago
> (http://code.google.com/p/playn/issues/detail?id=97), but as I was about to
> go finish the remaining bugs

Hi Joel,

I've been prototyping a game and testing it a lot with the DOM backend
on an iPhone 4. I've been seeing weird freezing behavior in the
display, and I've pared things down sufficiently to determine that
it's just the DOM that's not updating, rather than PlayN freezing.
Basically update() and paint() are getting called, but no visible
changes are taking place to the DOM for varying freeze durations up to
2-3000ms. It seems to be related to receiving touch events, since the
touch events are what trigger it.

I'm not particularly motivated to debug it further, since the DOM
backend can't go away soon enough, as far as I'm concerned. And the
problem does not happen with the new Canvas backend. I did put
together a simple demonstration:

http://samskivert.com/playn/touchtest-dom/
http://samskivert.com/playn/touchtest-canvas/

To see the failure, just click the button or click around anywhere on
the page, and it will periodically freeze. You can see that the game
update loop is not frozen by enabling debug logging in iOS Safari, and
you'll see that even when it's frozen, the JavaScript log messages
that are being sent ~10 times a second are accumulating.

Anyhow, that's all just FYI. What I really want to ask is whether
you've got fixes to the Canvas backend lying around uncommitted,
because I'm going to start looking into those bugs soonish if you
haven't. Also, I am wondering if you want to restructure your canvas
patch now that you want to preserve the DOM backend. Perhaps not,
since the DOM stuff should probably get new class names rather than
the Canvas stuff, which is more rightfully called, e.g.
HtmlGraphicsCanvas.java.

I'm aiming to make my prototype available to some testers this week,
so I want to sort out this pesky Canvas issue. But I think I can just
apply your canvas patch to a branch on the Three Rings PlayN
repository, and switch our build server to use that branch for now.
That way if you totally revamp the patch before ultimately pushing it
upstream, I won't have a big messy merge to do later.

I don't actually use rotation in my prototype at the moment, so the
existing canvas patch works just fine, but I'll still try to spend
some time looking into the rotation bugs this week just to help get
that code into the mainline sooner.

-- m...@samskivert.com

Joel Webber

unread,
Dec 19, 2011, 2:21:19 PM12/19/11
to Michael Bayne, pl...@googlegroups.com
Le 16 décembre 2011 12:45, Michael Bayne <m...@samskivert.com> a écrit :
On Fri, Dec 16, 2011 at 8:15 AM, Joel Webber <j...@google.com> wrote:
> Michael might have some opinion on this -- how much easier would it be,
> e.g., to port Spiral Knights to PlayN if its eventual 3D support used the
> nio interfaces?

The 3D library that we wrote for SK
(http://github.com/threerings/clyde) uses NIO in 42 source files (out
of 728), with the following breakdown:

 23 import java.nio.FloatBuffer;
 19 import java.nio.ByteBuffer;
 14 import java.nio.IntBuffer;
 11 import java.nio.ShortBuffer;
  6 import java.nio.DoubleBuffer;
  4 import java.nio.LongBuffer;
  4 import java.nio.CharBuffer;
  2 import java.nio.ByteOrder;
  1 import java.nio.charset.Charset;
  1 import java.nio.channels.FileChannel;

There's essentially no use of NIO in the main game code (modulo some
Steam integration that uses a combination of NIO and native code).

So it's probably not that big a deal. It would probably take a couple
of hours to change all the uses of NIO directly over to TypedArrays or
some other TypedArray-like buffer abstraction (maybe even less with
some judicious use of Perl).

Thanks, Michael, that's good information to have. I suppose in reality there just aren't all that many LWJGL or Android/Java GL games out there (the Android ones are mostly C++ iPhone ports, from what I hear). So I'll lean towards a simple TypedArray-like abstraction, on the assumption that a quick search/replace will cover most cases.

That said, something like SK is so complex that we would probably not
want the intermediating layer of PlayN and would instead just port
Clyde directly to Android or GWT+WebGL. Experimenting with the former
is on my long list of things to do. Though after seeing Chrome's
performance lately, maybe I should also experiment with the latter. :)

With the right changes, we can hopefully make it actually enticing to use the abstraction. But we'll see :)

Joel Webber

unread,
Dec 19, 2011, 2:25:15 PM12/19/11
to Michael Bayne, pl...@googlegroups.com, Matthew Mastracci
I don't have a device handy here to try it with, but hopefully Matt will have already seen these issues and have some ideas. If not, I'll try to take a look when I get home later tonight (where I have an iPad).

Anyhow, that's all just FYI. What I really want to ask is whether
you've got fixes to the Canvas backend lying around uncommitted,
because I'm going to start looking into those bugs soonish if you
haven't. Also, I am wondering if you want to restructure your canvas
patch now that you want to preserve the DOM backend. Perhaps not,
since the DOM stuff should probably get new class names rather than
the Canvas stuff, which is more rightfully called, e.g.
HtmlGraphicsCanvas.java.

I actually started resurrecting this patch over the weekend. I still have at least one transform bug, and one texture-repeat bug to sort out, but hopefully I'll have that sorted out today. I modified the patch to leave the dom backend around, but it makes canvas the default. I'll use this as the basis for fixing the canvas/layer stuff as soon as it's done.

I'm aiming to make my prototype available to some testers this week,
so I want to sort out this pesky Canvas issue. But I think I can just
apply your canvas patch to a branch on the Three Rings PlayN
repository, and switch our build server to use that branch for now.
That way if you totally revamp the patch before ultimately pushing it
upstream, I won't have a big messy merge to do later.

I don't actually use rotation in my prototype at the moment, so the
existing canvas patch works just fine, but I'll still try to spend
some time looking into the rotation bugs this week just to help get
that code into the mainline sooner.

Sounds good -- I'll let you know as soon as I have the patch back together for you to try out.

Matt Mastracci

unread,
Dec 19, 2011, 2:49:21 PM12/19/11
to Michael Bayne, Joel Webber, pl...@googlegroups.com
On 2011-12-19, at 11:57 AM, Michael Bayne wrote:

I've been prototyping a game and testing it a lot with the DOM backend
on an iPhone 4. I've been seeing weird freezing behavior in the
display, and I've pared things down sufficiently to determine that
it's just the DOM that's not updating, rather than PlayN freezing.
Basically update() and paint() are getting called, but no visible
changes are taking place to the DOM for varying freeze durations up to
2-3000ms. It seems to be related to receiving touch events, since the
touch events are what trigger it.

Two things to keep in mind when developing for iOS:

1) You have to cancel all touchstart/touchmove/touchend events. If you don't, Safari triggers the smooth scrolling logic, which stalls the rendering engine until the scroll process has finished.
2) Don't mutate the DOM at all within the touch event handlers. Queue up the events and re-run them on update(). Safari's rendering process can stall anywhere from 2-5s if you do too much DOM mutation inside of a touch event. 

There's also a weird bug on iPad where a touch event that starts on a DOM element that eventually disappears will fail to produce touchmove events after that. I've got a patch locally that puts up a <div> 'glasspane' overtop of the gameplay area to work around this, but I haven't had a chance to clean it up for commit yet.

Joel Webber

unread,
Dec 19, 2011, 2:58:44 PM12/19/11
to Matt Mastracci, Michael Bayne, pl...@googlegroups.com
Le 19 décembre 2011 14:49, Matt Mastracci <mat...@mastracci.com> a écrit :
On 2011-12-19, at 11:57 AM, Michael Bayne wrote:

I've been prototyping a game and testing it a lot with the DOM backend
on an iPhone 4. I've been seeing weird freezing behavior in the
display, and I've pared things down sufficiently to determine that
it's just the DOM that's not updating, rather than PlayN freezing.
Basically update() and paint() are getting called, but no visible
changes are taking place to the DOM for varying freeze durations up to
2-3000ms. It seems to be related to receiving touch events, since the
touch events are what trigger it.

Two things to keep in mind when developing for iOS:

1) You have to cancel all touchstart/touchmove/touchend events. If you don't, Safari triggers the smooth scrolling logic, which stalls the rendering engine until the scroll process has finished.

Would it be reasonable for the library to do this automatically, or would that likely screw up other interactions? Sure would be nice if we didn't have to keep giving this advice.

2) Don't mutate the DOM at all within the touch event handlers. Queue up the events and re-run them on update(). Safari's rendering process can stall anywhere from 2-5s if you do too much DOM mutation inside of a touch event. 

Did you ever run into this in the course of "normal" playn code? I *believe* all the regular code paths work hard not to touch dom/css during the update() phase -- all mutations should be deferred until paint(). The idea is that it should be ok to mutate layers in update() if you really want (or are just being lazy and/or not bothering with interpolation), but user-visible changes should only happen during paint().

Matt Mastracci

unread,
Dec 19, 2011, 3:45:41 PM12/19/11
to Joel Webber, Michael Bayne, pl...@googlegroups.com
On 2011-12-19, at 12:58 PM, Joel Webber wrote:

Le 19 décembre 2011 14:49, Matt Mastracci <mat...@mastracci.com> a écrit :
On 2011-12-19, at 11:57 AM, Michael Bayne wrote:

I've been prototyping a game and testing it a lot with the DOM backend
on an iPhone 4. I've been seeing weird freezing behavior in the
display, and I've pared things down sufficiently to determine that
it's just the DOM that's not updating, rather than PlayN freezing.
Basically update() and paint() are getting called, but no visible
changes are taking place to the DOM for varying freeze durations up to
2-3000ms. It seems to be related to receiving touch events, since the
touch events are what trigger it.

Two things to keep in mind when developing for iOS:

1) You have to cancel all touchstart/touchmove/touchend events. If you don't, Safari triggers the smooth scrolling logic, which stalls the rendering engine until the scroll process has finished.

Would it be reasonable for the library to do this automatically, or would that likely screw up other interactions? Sure would be nice if we didn't have to keep giving this advice.

I'd prefer to have PlayN do it automatically. If we call setPreventDefault(true) on the events returned by the touch handlers, the code will take care of this and user has the ability to undo it (with the caveat that it will probably screw everything up on Mobile Safari).

2) Don't mutate the DOM at all within the touch event handlers. Queue up the events and re-run them on update(). Safari's rendering process can stall anywhere from 2-5s if you do too much DOM mutation inside of a touch event. 

Did you ever run into this in the course of "normal" playn code? I *believe* all the regular code paths work hard not to touch dom/css during the update() phase -- all mutations should be deferred until paint(). The idea is that it should be ok to mutate layers in update() if you really want (or are just being lazy and/or not bothering with interpolation), but user-visible changes should only happen during paint().

The HTML DOM stuff is half immediate direct-mutation, half done post-paint(). Layer children are added/removed from the DOM immediately, setting the image of a layer is immediate, setting alpha is immediate, translation/rotation/visibility are handled in a method called after paint() is done.

I haven't seen any performance issues around this in update() or paint() (both of which are done in setTimeout callback on iOS). It seems like the touch event handlers are just 'magical'. I haven't had the time to dig into the problem to see if it's just the excessive DOM mutation, or if it's just taking too long to process the event-- even if it's just pure scripting/offscreen canvas manipulation-- that causes the problems. 

(as an aside: I have been bit by a slightly different problem as a result of this: when I add new layers to the tree as a result of a physics event, but don't position them until paint(), I sometimes get a sprite flash at (0,0) that shows the sprite in the default position)

I hadn't noticed this before, but the DOM code re-computes the CSS styles after each paint() call, whether they've changed or not. It might be worth investigating on my end whether a dirty flag would make this more performant. No need to re-compute CSS styles on layers that haven't been directly mutated between paint() calls. I don't know if WebKit is smart enough to avoid re-compositing if the current transformation matrix is just re-set via CSS properties.

Joel Webber

unread,
Dec 19, 2011, 5:01:35 PM12/19/11
to Michael Bayne, pl...@googlegroups.com, Matthew Mastracci
And.... it's out. See issue 109. I tested on Atlantis, and everything seems to work fine.
I'll come back and add more manual tests later, as I'm cleaning up the canvas/image layer stuff.

Michael Bayne

unread,
Dec 19, 2011, 5:31:27 PM12/19/11
to Matt Mastracci, Joel Webber, pl...@googlegroups.com
On Mon, Dec 19, 2011 at 11:49 AM, Matt Mastracci <mat...@mastracci.com> wrote:
> 1) You have to cancel all touchstart/touchmove/touchend events. If you
> don't, Safari triggers the smooth scrolling logic, which stalls the
> rendering engine until the scroll process has finished.

I am doing that.

> 2) Don't mutate the DOM at all within the touch event handlers. Queue up the
> events and re-run them on update(). Safari's rendering process can stall
> anywhere from 2-5s if you do too much DOM mutation inside of a touch event.

Perhaps this is the source of the problem, though my simple test case
just redraws a single button. Rerendering that button is apparently
enough to cause the delays.

-- m...@samskivert.com

Joel Webber

unread,
Dec 20, 2011, 10:29:44 AM12/20/11
to Matt Mastracci, Michael Bayne, pl...@googlegroups.com
Le 19 décembre 2011 15:45, Matt Mastracci <mat...@mastracci.com> a écrit :
On 2011-12-19, at 12:58 PM, Joel Webber wrote:
Le 19 décembre 2011 14:49, Matt Mastracci <mat...@mastracci.com> a écrit :
On 2011-12-19, at 11:57 AM, Michael Bayne wrote:

I've been prototyping a game and testing it a lot with the DOM backend
on an iPhone 4. I've been seeing weird freezing behavior in the
display, and I've pared things down sufficiently to determine that
it's just the DOM that's not updating, rather than PlayN freezing.
Basically update() and paint() are getting called, but no visible
changes are taking place to the DOM for varying freeze durations up to
2-3000ms. It seems to be related to receiving touch events, since the
touch events are what trigger it.

Two things to keep in mind when developing for iOS:

1) You have to cancel all touchstart/touchmove/touchend events. If you don't, Safari triggers the smooth scrolling logic, which stalls the rendering engine until the scroll process has finished.

Would it be reasonable for the library to do this automatically, or would that likely screw up other interactions? Sure would be nice if we didn't have to keep giving this advice.

I'd prefer to have PlayN do it automatically. If we call setPreventDefault(true) on the events returned by the touch handlers, the code will take care of this and user has the ability to undo it (with the caveat that it will probably screw everything up on Mobile Safari).

Will it just stop you from uselessly touch-scrolling the browser window? If so, that actually sounds like a good thing. It's really disconcerting when you're able to "drag" a browser window in which dragging serves no purpose.
2) Don't mutate the DOM at all within the touch event handlers. Queue up the events and re-run them on update(). Safari's rendering process can stall anywhere from 2-5s if you do too much DOM mutation inside of a touch event. 

Did you ever run into this in the course of "normal" playn code? I *believe* all the regular code paths work hard not to touch dom/css during the update() phase -- all mutations should be deferred until paint(). The idea is that it should be ok to mutate layers in update() if you really want (or are just being lazy and/or not bothering with interpolation), but user-visible changes should only happen during paint().

The HTML DOM stuff is half immediate direct-mutation, half done post-paint(). Layer children are added/removed from the DOM immediately, setting the image of a layer is immediate, setting alpha is immediate, translation/rotation/visibility are handled in a method called after paint() is done.

I haven't seen any performance issues around this in update() or paint() (both of which are done in setTimeout callback on iOS). It seems like the touch event handlers are just 'magical'. I haven't had the time to dig into the problem to see if it's just the excessive DOM mutation, or if it's just taking too long to process the event-- even if it's just pure scripting/offscreen canvas manipulation-- that causes the problems. 

(as an aside: I have been bit by a slightly different problem as a result of this: when I add new layers to the tree as a result of a physics event, but don't position them until paint(), I sometimes get a sprite flash at (0,0) that shows the sprite in the default position)

I hadn't noticed this before, but the DOM code re-computes the CSS styles after each paint() call, whether they've changed or not. It might be worth investigating on my end whether a dirty flag would make this more performant. No need to re-compute CSS styles on layers that haven't been directly mutated between paint() calls. I don't know if WebKit is smart enough to avoid re-compositing if the current transformation matrix is just re-set via CSS properties.

Damn it, you're right (side note: I hate trying to keep two scene graphs in sync; it's just nasty). If we really *had* to, we could batch up all DOM mutations, though that could get a bit ugly. Please let me know if you find any evidence (or even suspect) that the lack of batching is causing problems.

Matt Mastracci

unread,
Dec 20, 2011, 12:07:59 PM12/20/11
to Joel Webber, Michael Bayne, pl...@googlegroups.com
On 2011-12-20, at 8:29 AM, Joel Webber wrote:

Le 19 décembre 2011 15:45, Matt Mastracci <mat...@mastracci.com> a écrit :
I'd prefer to have PlayN do it automatically. If we call setPreventDefault(true) on the events returned by the touch handlers, the code will take care of this and user has the ability to undo it (with the caveat that it will probably screw everything up on Mobile Safari).

Will it just stop you from uselessly touch-scrolling the browser window? If so, that actually sounds like a good thing. It's really disconcerting when you're able to "drag" a browser window in which dragging serves no purpose.

It will stop that useless scrolling. There's really no real reason for not cancelling touch events by default inside the root element or on the page if part of a drag initiated from inside the root element. 

I'll throw together a patch for this and selecting DOM mode on iPad/iPhone in autodetect mode.

The HTML DOM stuff is half immediate direct-mutation, half done post-paint(). Layer children are added/removed from the DOM immediately, setting the image of a layer is immediate, setting alpha is immediate, translation/rotation/visibility are handled in a method called after paint() is done.

Damn it, you're right (side note: I hate trying to keep two scene graphs in sync; it's just nasty). If we really *had* to, we could batch up all DOM mutations, though that could get a bit ugly. Please let me know if you find any evidence (or even suspect) that the lack of batching is causing problems.

Will do. 

Matt Mastracci

Michael Bayne

unread,
Dec 20, 2011, 12:12:03 PM12/20/11
to Matt Mastracci, Joel Webber, pl...@googlegroups.com
On Tue, Dec 20, 2011 at 9:07 AM, Matt Mastracci <mat...@mastracci.com> wrote:
> I'll throw together a patch for this and selecting DOM mode on iPad/iPhone
> in autodetect mode.

Can we not make this the default? My prototype performs much better
using the Canvas backend on iPhone/iPad Mobile Safari. Probably our
games differ in their performance characteristics, but I think it's
not a guarantee that the DOM backend is preferable on iOS. Plus the
Canvas backend is already "more correct" than the DOM backend.

Perhaps we can allow this use of DOM on iOS to be configured on HtmlPlatform?

-- m...@samskivert.com

Michael Bayne

unread,
Dec 20, 2011, 12:12:34 PM12/20/11
to Matt Mastracci, Joel Webber, pl...@googlegroups.com
On Tue, Dec 20, 2011 at 9:12 AM, Michael Bayne <m...@samskivert.com> wrote:
> Can we not make this the default?

Sorry, by "this" I mean defaulting to DOM on iOS. I'm fine with
automatically preventing default on touch events.

-- m...@samskivert.com

Matt Mastracci

unread,
Dec 20, 2011, 12:31:24 PM12/20/11
to Michael Bayne, Joel Webber, pl...@googlegroups.com
Are you running an iPad 2 and/or iOS5, or making use of surface layers? I know for sure that canvas runs like molasses on iPad1/iOS4 - it doesn't seem to be capable of running faster than 2-5 fps for an extremely trivial game. The same game is reasonably smooth in Canvas/iPad2/iOS5, however. 

I haven't tested iPad1/iOS5, but we might be able to switch to DOM mode for iOS4 only if iPad1/iOS5 performs well. I've looked for a way to detect iPad1 vs. iPad2, but this appears to be impossible.

I'd also be OK with a switch behind the scenes to control this auto-detection behavior.

Michael Bayne

unread,
Dec 20, 2011, 2:57:02 PM12/20/11
to Matt Mastracci, Joel Webber, pl...@googlegroups.com
On Tue, Dec 20, 2011 at 9:31 AM, Matt Mastracci <mat...@mastracci.com> wrote:
> Are you running an iPad 2 and/or iOS5, or making use of surface layers? I
> know for sure that canvas runs like molasses on iPad1/iOS4 - it doesn't seem
> to be capable of running faster than 2-5 fps for an extremely trivial game.
> The same game is reasonably smooth in Canvas/iPad2/iOS5, however.

Oh, yeah, this is on iOS5, which has hardware accelerated Canvas. I
have an iPad 1 as well (running iOS 5), so I'll test on that when I
get home, to be sure that things perform reasonably.

I'm not averse to defaulting to DOM on iOS4, though the number of
people that ship a PlayN game for iOS4 using the HTML5 backend may
never exceed one. I certainly aim to have a native iOS solution by
hook or crook before I ship the game I'm working on now.

Are you planning to ship your game PhoneGap-style, or is it going to
be deployed via the web?

-- m...@samskivert.com

Matt Mastracci

unread,
Dec 20, 2011, 3:24:16 PM12/20/11
to Michael Bayne, Joel Webber, pl...@googlegroups.com
On 2011-12-20, at 12:57 PM, Michael Bayne wrote:

On Tue, Dec 20, 2011 at 9:31 AM, Matt Mastracci <mat...@mastracci.com> wrote:
Are you running an iPad 2 and/or iOS5, or making use of surface layers? I
know for sure that canvas runs like molasses on iPad1/iOS4 - it doesn't seem
to be capable of running faster than 2-5 fps for an extremely trivial game.
The same game is reasonably smooth in Canvas/iPad2/iOS5, however.

Oh, yeah, this is on iOS5, which has hardware accelerated Canvas. I
have an iPad 1 as well (running iOS 5), so I'll test on that when I
get home, to be sure that things perform reasonably.

Great! I'd be interested to know if it's "iPad 1" or "iOS4" that causes the brutal performance in canvas mode. The inability to downgrade iDevices from iOS[n] to iOS[x<n] is a big frustration for me - I can only test iPad1 in iOS4, or iPad2 in iOS5.

I'm not averse to defaulting to DOM on iOS4, though the number of
people that ship a PlayN game for iOS4 using the HTML5 backend may
never exceed one. I certainly aim to have a native iOS solution by
hook or crook before I ship the game I'm working on now.

Are you planning to ship your game PhoneGap-style, or is it going to
be deployed via the web?

Initially via the web, but (possibly) PhoneGap-style in the future. I need to dig into the JS JIT situation more before then: I can't recall whether Apple started accelerating JS on iOS4 or 5. If we're currently getting acceleration on iOS4 and we lose it by using a PhoneGap wrapper, I might not be able to go that route.

Joel Webber

unread,
Dec 20, 2011, 3:32:47 PM12/20/11
to Matt Mastracci, Michael Bayne, pl...@googlegroups.com
Le 20 décembre 2011 15:24, Matt Mastracci <mat...@mastracci.com> a écrit :
On 2011-12-20, at 12:57 PM, Michael Bayne wrote:
On Tue, Dec 20, 2011 at 9:31 AM, Matt Mastracci <mat...@mastracci.com> wrote:
Are you running an iPad 2 and/or iOS5, or making use of surface layers? I
know for sure that canvas runs like molasses on iPad1/iOS4 - it doesn't seem
to be capable of running faster than 2-5 fps for an extremely trivial game.
The same game is reasonably smooth in Canvas/iPad2/iOS5, however.

Oh, yeah, this is on iOS5, which has hardware accelerated Canvas. I
have an iPad 1 as well (running iOS 5), so I'll test on that when I
get home, to be sure that things perform reasonably.

Great! I'd be interested to know if it's "iPad 1" or "iOS4" that causes the brutal performance in canvas mode. The inability to downgrade iDevices from iOS[n] to iOS[x<n] is a big frustration for me - I can only test iPad1 in iOS4, or iPad2 in iOS5.

I have an iPad1 here at the office that's currently on iOS4. If you can give me a URL to try, I'd be happy to try it on 4, upgrade it to 5, then try it again.
I'm not averse to defaulting to DOM on iOS4, though the number of
people that ship a PlayN game for iOS4 using the HTML5 backend may
never exceed one. I certainly aim to have a native iOS solution by
hook or crook before I ship the game I'm working on now.

Are you planning to ship your game PhoneGap-style, or is it going to
be deployed via the web?

Initially via the web, but (possibly) PhoneGap-style in the future. I need to dig into the JS JIT situation more before then: I can't recall whether Apple started accelerating JS on iOS4 or 5. If we're currently getting acceleration on iOS4 and we lose it by using a PhoneGap wrapper, I might not be able to go that route.

I'm pretty sure they've had a JITing version of JSCore on there for a while, and they you always lose the JIT in apps other than Safari, because it's one of the only ones whitelisted to be able to create executable, writable pages.
Reply all
Reply to author
Forward
0 new messages