Integration of PulpCore in swing : first shot.

[nouknouk]

Hi everybody.

I wondered to be able to easily integrate pulpcore engine in any Swing
UI.
Therefore i just began to implement one or two classes to do it.

The objective is to be able to have something simple as that:

public void myFunc() {
JFrame frame = new JFrame();
PulpPanel panel = new PulpPanel("mypackage.mySceneclass");
frame.add(panel);
frame.setVisible(true);
}


A sample (web start) here : http://nouknouk.free.fr/bazar/pulpcore-jpanel.jnlp
The sources are available here : http://nouknouk.free.fr/bazar/pulpcore-jpanel-src.zip

Now some comments:

- I am anything but an expert in Swing. So most of the things coded
are probably not pertinent. Thoses sources are closer to a 'proof of
concept' than anything final.

- my philosophy was to *not* modify anything in orginal pulpcore's
code. So only new classes have been added.

- the performance is just acceptable, because the buffer technology
used is image buffer, and the whole scene is repainted in the
component (because of artifacts encountered while resizing with other
methods).

- some things are supported:
* debug widget
* dirty rectangles
* leightweight component allowing menus to be painted over the
component)
* compilation without any custom building rules. I mean you can
integrate those source in any (netbeans) project, compile, and it
works.

Most features are not working. Notably, I experience issues with :

* Assets not tested, but should not work.

* user data is not persistent

* the resizing of the component and painting the unused space around
the scene (in case, for example, of a scene with a size ratio
(AUTO_FIT) and a component with a different one).

* Changement of the scene after JPanel's initialization.

* poor performances compared to original's code.


My main concern currently is about the Stage class : indeed I would
like to be able to do things outside of the animation thread (eg. the
main thread). The problem is that the Stage only has got getters - eg.
getScene() - that are static and rely on the current thread.

@ David : I suppose things portions of pulpcore's original code could
be factorized, and some other things would help supporting pulpcore
swing's integration. For example, having something like non-static
function to access members of Stage object would be helpful. Would you
be ok to discuss about some little things like that ?

Any comments, improvements, ... are welcome :-)

[David Brackeen]

Wow, good job. I wasn't able to launch the jnlp - looks like the codebase is set to a local path on your machine - but I'm impressed anyway.

As for Stage, I'm open to ideas on different ways to get the AppContext, but those methods aren't thread-safe - they need to be called from the animation thread anyway, to prevent possible conflict with the animation loop. 

I'm also open to other changes - I've know many under-the-hood changes will have to happen to port to other platforms (especially Android or LWJGL). So let me know any ideas you have.

Are you building off of the 0.10 branch or the trunk?

Assets may actually work since it loads resources from the jar if there is no zip.

Hmm I'm wondering what could be done to increase the performance. It might be useful to offer the option of lightweight of heavyweight PulpPanel.

As soon as we get the wrinkles ironed out I'll add it to the main distribution!

[nouknouk]

> Wow, good job. I wasn't able to launch the jnlp - looks like the codebase is
> set to a local path on your machine - but I'm impressed anyway.

Ok, fixed. Sorry for that.

> As for Stage, I'm open to ideas on different ways to get the AppContext, but
> those methods aren't thread-safe - they need to be called from the animation
> thread anyway, to prevent possible conflict with the animation loop.

I understand thread safety is mandatory.
The design of pulpcore's is really good for the purpose it is aim to,
i mean applets. Indeed, with applets, changes are initiated by events
fired by the pulpcore's engine itself (mouse clicked, moved, ...).

But in case of Swing's integration, many other thread coulds be
involved and fire 'events' that will interact with pulpcore, like:
- Swing's Event Dispatch Thread (EDT)
- 'main' thread
- the case of multiple 'PulpPanels' running in parallel in the same
application
- developper's custom threads. For example I've got a network
framework that fire events when it receives a message. The event is
processed in socket's dedicated thread.

Thus, being able to explicitely execute something inside one
PulpPanel's thread would be uselful. I mean being able to invoke
easily 'Scene2D::invokeAndWait()', like we do for swing's interaction
and EDT.

The current problem is that given an application context, you cannot
access its associated Scene2D object : the only (public) way currently
implemented is Stage::getScene(). But this function is a static one
and relies on the fact that getScene() will be called from the right
thread.

One solution could be to keep only one static function -
getThisStage() - and transform other function in non-static ones. The
'static capability' would be still available by calling '
getThisStage().getScene() ' while ' getScene() ' alone would provide
the non-static getter.

NOTE: I experienced another thing that may be annoying in the future :
any scenegraph's change inside a Scene (adding a Group, ...) is
forbidden if you are not in the animation thread. This is essential
when the scene is the one currently displayed by the animation thread.
But if the scene is *not* attached to any app context, allowing its
changes from any thread would be pratical and not dangerous (afaik).
For example, It would let do things like that:

public void myFuncCalledFromMainThread() {
// an already existing PulpCoreJPanel
PulpPanel panel;

// we instanciate a new scene:
Scene2D scene = new Scene2D();

// problem just below : adding a Group is forbidden from main
thread.
scene.add(new Group());

// the goal : being able to attach the new scene to the panel.
panel.setScene(scene);

}

>
> I'm also open to other changes - I've know many under-the-hood changes will
> have to happen to port to other platforms (especially Android or LWJGL). So
> let me know any ideas you have.

Actually, a few help from you would be really helpful for me, because:
- i'm not experienced with java graphic's framework at all.
- you know well your framework and all the pitfalls you faced while
implementing it.
On the other hands, I'm aware:
- integration in swing is not the original goal of your framework, and
thus may not be a priority for you
- you probably have other (more interresting) things to do than
helping me.

>
> Are you building off of the 0.10 branch or the trunk?
>

Good question. I got your sources from SVN (pulpcore-read-only),
probably the trunk (but not sure).

> Assets may actually work since it loads resources from the jar if there is
> no zip.

Indeed, yes. My apologizes : that's already used at least for loading
the system font (PNG included in the codebase folder of the sources
provided above).

>
> Hmm I'm wondering what could be done to increase the performance. It might
> be useful to offer the option of lightweight of heavyweight PulpPanel.

At all :-)

>
> As soon as we get the wrinkles ironed out I'll add it to the main
> distribution!

For the moment, my code is just ... crappy coded (the other name of
'proof of concept' in my mouth ;-)


Other general ideas, as they come:

- moving functionnalities that are 'applet-specific' inside the
applet's dedicated classes (userData, talkbackURL,
getBrowserName, ...)

- factorize common code between applet classes and jpanel ones
(Corexxx, xxxAppContext, xxxInput, xxxPlatform) and maybe implement
use common interfaces instead of specific implementation in other
parts of the framework.

Best regards,

Nouk²

[Juan José Medina]

Excelent work!!!!

[David Brackeen]

Hmm, after thinking about it, the only thing other threads need is an invokeLater() for the AppContext. So I'm thinking I'll move invokeLater() from Scene2D to Stage (where the animation thread is). 

Since an AppContext keeps a reference to its Stage instance, that means PulpPanel could also easily have an invokeLater() wrapper. So the example would become:

public void myFuncCalledFromMainThread() {

    // Internally calls stageInstance.invokeLater()

    pulpPanel.invokeLater(new Runnable() {

        public void run() {

            // In the animation thread now!

            // we instanciate a new scene:

            Scene2D scene = new Scene2D();

             scene.add(new Group());

            Stage.setScene(scene);

        }

    });

}

What do you think? This would also be useful for PulpCore Player - so far it cant have decent menu items because of the thread issue.

As for user data, you can use the Persistence API pretty easily (it doesn't work in applets but for apps it's fine).

I'm mildly familiar with low-level Swing stuff, so I might be able to help with any graphic issues that come up.

[nouknouk]

David,

I've just got a new version for testing the integration of my non-
static's Stage idea.

the sample: http://nouknouk.free.fr/bazar/pulpcore-jpanel2.jnlp
(note the interaction with the swing UI by clicking on the Swing's
button)

the sources: http://nouknouk.free.fr/bazar/pulpcore-jpanel2-src.zip
Here are the full sources (i mean the sample + PulpJPanel + changes in
pulpcore's sources).

- The change I did is about my idea to transform all Stage's static
functions into non-static ones. I had to modify the Stage class itself
and several dependencies spread over several other classes.

- I had to add a Stage pointer in the Scene class, representing the
stage that owns the scene ( == null when scene is instanciated ; set
automatically when stage.setScene() pushScene(), ... functions are
called).
Only two static functions remains : getThisScene() and getThisStage().
Their return value depends on the thread that called them.

No regression tests have been made at all. Especially, I have (really
serious) doubts about the Applet platform.

I get:

- stage's functions that are not thread dependant (like pushScene(),
getHeight(), ...) can be called from any thread (included the
animation thread of another PulpCorePanel).

- the stage pointer inside Scene (and thus Scene2D) still give a way
to get usefull information in Scene2D.loading() (like width,
height, ...).

- the instanciation of a Scene2D could be made from anywhere, in any
thread.

- changing the current scene of a panel is now possible.


Let me know about your feedback.



About your solution, I'll deeper investigate ... but for now, it's
time to sleep (as well as it was already 4 hours ago ;-)


Best regards,

Nouk²

[nouknouk]

On 24 jan, 03:25, "David Brackeen" <brack...@gmail.com> wrote:

> Hmm, after thinking about it, the only thing other threads need is an
> invokeLater() for the AppContext. So I'm thinking I'll
> move invokeLater() from Scene2D to Stage (where the animation thread is).

[...]

> What do you think? This would also be useful for PulpCore Player - so far it
> cant have decent menu items because of the thread issue.

Hi David,

First of all, the following is just my point of view that aims to be
discussed.
Please don't be offended if some sentences look dogmatic. It's not
their goal at all.

Let's go about my global feelign about the threading discussion:

I have a personal preference for avoiding as much as possible
restricting the usage of the framework to specific threads. The less
restrictive the framework is, the better it would be.
In other words, the design of the framework should only prevent its
usage in other threads when it is critical. It is (afaik) only when a
scene is attached to an animation thread.

I'm not really clear, but the Swing framework's principles (with its
EDT) is for me a good solution:
- you can do whatever you want from any thread: instanciate objects,
perform things on them, ...
- you have tools for handling the thread concurrency issues
(SwingUtilites.invokexxx()
- it's the developper's responsability to avoid concurrency issues,
with the provided tools.

Applied to pulpcore's framework, it would mostly mean that:
- the model's classes (Scene2D, ...) could be instanciated and used in
any thread ... as long as the scene is not attached to an animation
thread.
- in addition one or two debug classes could be coded in odrer to help
developpers to find issues in their code.

But such changes in the framework cannot be done without several
changes in the framework's interface. Thus, those changes would break
the compability with the existent. That's the main ptifall.

On the other hand: I'm quite convinced this would lead to a more
robust framework, more ready for further improvements and portage.
Moreover, sticking to the swing's philosophy would be good for
developpers that discover pulpcore. Indeed, their understanding of the
'good practies' in pulpcore would be immediate, if they have already a
basic knowledge of Swing's principles. And probably it would be an
argument to make them adopt pulpcore.

It's up to you to decide if - at current developpement's phase -
breaking the compatiblity with previous versions of the framework is
conceivable or not.
If it is, I would take time to perform a full review of the
framework's architecture and give a list of concrete suggestions.

Best regards,

Nouk²

[nouknouk]

Second point: About the usage of static functions in several classes
(especially CoreSystem and Stage) :

Same remark as above, the following is just my point of view that aims

to be
discussed. Please don't be offended if some sentences look dogmatic.
It's not
their goal at all.

My feeling is that current usage of static functions was appropriated
as long as the framework is intended to develop applets. Indeed, in
that case, there is always only one Scene displayed at a time. Thus,
there is only one animation thread and all the processing is initiated
by event coming from this thread.

But in the case of pulpcore's integration in Swing, the context
changes, because:
- code's execution happens in several thread (main thread, EDT thread,
pulp's animation thread, other custom threads)
- the framework must handle the usage of several scenes at a time.

That's why I think the static functions will become a problem. Indeed:

- most of them retrive - in backstage - the context according to the
current thread. This can result in a mis-understanding for the
developper, as their name don't mention explicitely they are thread
dependant.

- moreover, if some of the proposed changes made above are done (ie.
my threading thought), relying on the current thread loose its sense,
as the framework would be aimed to be used from any thread.

- finally, the usage of singleton (static functions are close to
singleton's pattern) is often seen as an anti-pattern, mostly because
it freezes further changes (derivation of the singleton class, further
need for multiple instance of this class, ...).


Thus, I think the functions that rely on current thread should be
removed as much as possible.


I did some list of the current key classes that are needed in several
places : Platform, Input, AppContext, SoundEngine, Surface, Stage,
Scene, CoreApplet (and their respective derivated classes).

In those, we can see that :
- CoreSystem (singleton) holds the unique reference to the Platform.
- Platform holds a reference on the SoundEngine
- AppContext holds references on CoreApplet, Input, Surface, Stage
- Stage holds references on the list of scenes, Input, AppContext (its
manager), Surface & the animation thread.

That means, given an access to the instances of Platform and
AppContext, all other key parts of the framework can be accessed.

That's why, to my mind, only a few static functions could be kept,
mostly for convenience. I think too that when a function's result
depends on the current thread, its should explicitely mention it. (I
like the current 'getThisxxx()' naming). The two staitc functions
remaining could be:

- AppletPlatform.getInstance() and PlupPanelPlatform.getInstance()
(this is a singleton and even further needs won't involve the need of
multiple instances).

- AppContext.getThisAppContext() : for convenience (but its use would
be discouraged). To easily get the context associated to a thread.


To achieve that, we need to replace those functions by non-static
access in key places of the framework. I propose:

- to add a Scene2D pointer in each class of the package
pulpcore.sprite. That means each sprite will belong to one specific
Scene2D. In that case, the constructor of the sprite should take at
least a Scene pointer as parameter.

- to add a Context pointer in Scene2D, to be able to retrieve the
context owning this scene. As porposed in my previous code poted
above, I think a Scene2D could belong to *zero* or one Context at a
time.

More globally: As for the threading toughts, such changes would break
the compatibility with previous versions of pulpcore, but I think it
would be a great enhancement of the framework and make it even more
robust and flexible.
If my suggestions look no too bad for you, I could manage such change
and propose concrete things.

What do think of those ideas ?

I've got other ideas about moving a few responsabilities from one
class to another, but it's minor compared to the two suggestions
above.

Hope my (too) long posts are not too much boring to read, but it's
quite hard to propose changes and their reasons without being precise.

Best regards,

Nouk²

[jequi]

David is obviously the one who will answer Nouk2; however, it brings
to question, what are the performance implecations that come with
these proposed changes?

[David Brackeen]

I understand your concerns. Thanks for posting, I appreciate all the thought you've put into this.

To sum it up, the main difference between threading on Swing and PulpCore is that PulpCore's threading is enforced, while Swing's is optional (the developer just has to know which operations in the EDT). And PulpCore's static methods in CoreSystem, Input, and Stage (kind of pseudo-singletons - one instance per app) only work if the developer calls them from the animation thread. 

There are two things I believe I could do now, in the 0.11 version:

1) Have invokeXXX methods that could be called from any thread, and don't depend on a Scene2D.

2) Don't enforce threading rules in scene.add()/remove() or group.add()/remove() unless the scene is the current active scene.

After added the two features above, PulpCore will behave more like Swing.

The other changes you mention I'll have to put some thought into. I would like to keep the API as simple as possible, with the idea that the developer doesn't have to use any of the "under the hood" classes in the pulpcore.platform.* package. I like some of your ideas, but what would the API look like for developers?

The other problem is a lot of those methods in CoreSystem, Input, and Stage aren't thread safe. For example, Stage.pushScene() needs to be called on the animation thread anyway.

[nouknouk]

On 25 jan, 21:09, jequi <shashepp...@gmail.com> wrote:

> David is obviously the one who will answer Nouk2; however, it brings
> to question, what are the performance implecations that come with
> these proposed changes?

The answer is none

The purpose deals only with refactoring the global design of the
classes.
The code itself, especially the one for rending would remain exaclty
the same.
If you prefer, it's just about keeping the same code, but moving parts
from one place to another, to gain flexibility and capacity to change
(for further improvements).

As we should never say never, it could be a very ... very little
impact.
But not only it is really negligible (somehting like one or two
pointer's assignement), but also the "loss" will only happen at
framework initialization and maybe when you change the scenegraph's
content, which doesn't happend at each frame at all.

Promised ;-)

Best regards,

Nouk²

[nouknouk]

On 25 jan, 21:22, "David Brackeen" <brack...@gmail.com> wrote:

> To sum it up, the main difference between threading on Swing and PulpCore is
> that PulpCore's threading is enforced, while Swing's is optional (the
> developer just has to know which operations in the EDT). And PulpCore's
> static methods in CoreSystem, Input, and Stage (kind of pseudo-singletons -
> one instance per app) only work if the developer calls them from the
> animation thread.

About calling Input functions, you're right, i missed its change
happen at each frame's update. But as its interface is mainly composed
of getters, I d'ont think calling them from another thread would be
critical (i mean here "can lead into some kind of concurrency
exception) even if it could return erroneus results. Isn't it ?

About unicity of Input and Stage, It won't be the case anymore as soon
as we have more than one scene displayed per application. In fact,
those instances are actually unique for Applet, but it has to be
handled for swing's integration, as each PulpPanel will has its own
input and stage attached to its Component.
A good example : if a click happens, it will only have to be
propagated inside the right PulpPanel and not the other running.
You can check my sources of PulpPanel : it's the implementation i'm
already using : there's one appContext per PulpPanel, which holds its
owns Input and Stage instances.

>
> There are two things I believe I could do now, in the 0.11 version:
> 1) Have invokeXXX methods that could be called from any thread, and don't
> depend on a Scene2D.
> 2) Don't enforce threading rules in scene.add()/remove() or
> group.add()/remove()
> unless the scene is the current active scene.
>

Thanks a lot, it would solve a blocking issue on my side.

About other things, i assume it would be a bigger work, and you
probably have other things to do. Maybe you even have a real life in
the real world ;-)

But If we decide to do such thing, that's something I can do on my
side : refactoring with a recent IDE looks so smart for me compared to
the same in (my first love) C++ ;-)

> After added the two features above, PulpCore will behave more like Swing.
>
> The other changes you mention I'll have to put some thought into. I would
> like to keep the API as simple as possible, with the idea that the developer
> doesn't have to use any of the "under the hood" classes in the
> pulpcore.platform.* package. I like some of your ideas, but what would the
> API look like for developers?

As I propose to add a Scene pointer in each Sprite (and a context
pointer in each Scene), there are simple solutions to get a
"developper-friendly" API : you can provide convenience (and not
static) getters inside the Sprite's public interface (can be in Scene
too).

eg: Sprite.getInput() { return
privateScenePointer.getAppContext().getInput(); }

Thus the API will be as simple as possible : all the accessible
features will be directly found inside the interface of the Sprite
class. The developper won't have to bother with more 'under the hood'
classes if he don't need them. But he will have too a better and more
comprehensible access to them for specific needs.

About the animation's processing and timelines, it would be just a
little bit more fussy to implement, but the principle would remain the
same.

>
> The other problem is a lot of those methods in CoreSystem, Input, and Stage
> aren't thread safe. For example, Stage.pushScene() needs to be called on the
> animation thread anyway.
>

Two solutions :

- first one is to say "it's the responsability of the developper" to
have readen the javadoc and to encapsulate such call inside an
'invokexxx()' call. This sticks to the Swing's philosophy.

- the other solution is to assist the developper by adding a check
inside the pushScene() function to check if the thread is the
animation one. If it is not, the call can be automatically re-routed
to an invokeAndWait() version. Basically, something like that:

public void pushScene(Scene2D scene) '
if (currentThread == thisStageThread) {
// do the processing.
}
else {
this.invokeAndWait(new Runnable() {
pushScene(scene);
}
}


Last question, in two words : did you ever tried to run multiple
pulpcore's applets inside the same web page ? Do they share the same
sandbox and classes' code ?
I'm just thinking about a possible issue with Singletons if it's the
case ... but you probably already take that in account.

Ok, i lied, there was more than two words... as usual... ;-)


Best regards,

Nouk²

[David Brackeen]

The first part of this is done, in SVN: Scene2D and Group are now thread-safe. No more ConcurrentModificationExceptions! Also, sprites can remove themselves from their parent group.

CoreSystem, Input, Stage, Properties, Timelines and Sprites aren't thread-safe - so continue to use scene2D.invokeLater() when modifying those. Also, Sprite now has a getScene2D() method.

I've got one more "big change" to do before releasing 0.11 - converting the rendering to use premultiplied alpha. It will speed up some rendering operations, and is required for rendering onto transparent surfaces without getting crazy math (Groups will eventually have an option of a back-buffer, which will allow some new kinds of effects).

[nouknouk]

Huge work !

Thanks for all those changes.
I'll check them as soon as possible (my work occupies me really too
much those days).

Nouk².

On 9 fév, 23:30, "David Brackeen" <brack...@gmail.com> wrote:
> The first part of this is done, in SVN: Scene2D and Group are now
> thread-safe. No more ConcurrentModificationExceptions! Also, sprites can
> remove themselves from their parent group.
> CoreSystem, Input, Stage, Properties, Timelines and Sprites aren't
> thread-safe - so continue to use scene2D.invokeLater() when modifying
> those. Also, Sprite now has a getScene2D() method.
>
> I've got one more "big change" to do before releasing 0.11 - converting the
> rendering to use premultiplied alpha. It will speed up some rendering
> operations, and is required for rendering onto transparent surfaces without
> getting crazy math (Groups will eventually have an option of a back-buffer,
> which will allow some new kinds of effects).
>