Sharing between frameworks (Nme / OpenFl / Kha / GameDuell / Heaps / Snow / YourGreatFramework )

[Hugh]

We all know that, without a doubt,  the Apis / Workflows / Build tools for our frameworks are best - even if Nme did not *even make the list* of Roberts framework coolness ranking.

That said, there are a lot of shared concepts (in the case of Nme, OpenFl and GameDuell, there is a lot of shared syntax) in the framework tools.  So it seems to me that we should be able to share a whole bunch of code, while still vehemently disagreeing over whether you use signals or events, sdl or openal, xml or json, flash or a custom draing api.

I am thinking here originally about native extensions, since "pure haxe" libraries can already be shared relatively easily.  However any discussions on this point would be welcome.

When it down to it, an extension will have:

1. haxe code

2. native code (cpp/java/objc/c#)

    - maybe precompiled into ndll/.lib

3. native dependencies (libraries/jars/frameworks)

4. native hooks (eg, call something on startup/shutdown)

5. assets / data to embed

6. project settings (eg manifest requirements)

While a particular framework will relocate and compile these various bits to get it all working together, the data in points 1-6 will essentially be the same.

So what I am proposing is defining a framework-neutral specification or a config file plus a directory layout for this data.  The idea is that the individual framework tools can either convert this specification into its native format (.nmml/KhaMake whatever), or read the spec directly.  It may also require some code-gen (eg, generate hooks, maybe JNI calls).

A slight variation on this would be add an additional "frameworks" directory where other frameworks can pull-request their custom build-files, but I think that this could be more work.

To be acceptable to all frameworks, I think the config needs to be parsable by haxe code without a library (or agree on library, which has proved problematic) - which gives xml, json or some simple line-based ascii.  I'm leaning towards json, or xml if we allow the "if" style conditionals.  In the json case, I suggest multiple files like "common" "android" and "android-x86" which get ready sequentially and join together - but again open to ideas.

JNI calls is a tricky one - does anyone have a haxe->java jni solution that is not derived from the "HaxeObject" nme jni code?  Maybe this could itself be made into library, of maybe abstracted into framework-neutral Api, which can be relocated via a typedef (eg, "framework.Jni" gets mapped to "nme.JNI" or whatever)

Finally, I suggest moving the android extensions from ant to gradle, inline with google's move.  This might break a few things, but I think we need to all move in this direction.

Any comments?

Hugh

[Robert Konrad]

Let's get right into it. I usually start with the most complicated thing. In Kha that's the C/C++ build system. It uses actual JavaScript code files for that, which is useful, but maybe we can do without that. What remains would be a file that contains include-dirs, #defines, and a list of patterns which define the files to compile. Kha doesn't currently natively support any kind of precompiled libraries. I strongly prefer source libs, because it's important to me that everybody who uses Kha can always step though all of the code. But on the other hand, we should probably just support both, including a lib isn't that complicated after all. Do other frameworks have any mechanisms for supporting C/C++ source code packages or is it all ndlls?

PS: I'm ok with JSON. I like Mr Crockford.

[Tarwin Stroh-Spijer]

Awe Hugh I still think you're cool!

But in all seriousness, sounds like good plans.

Tarwin Stroh-Spijer
_______________________

phone: +1 650 842 0920

Developer at Fanplayr Inc. (Palo Alto)

Original at Touch My Pixel (touchmypixel.com)
_______________________

On Fri, Aug 7, 2015 at 2:26 AM, Robert Konrad <rob...@gmail.com> wrote:

Let's get right into it. I usually start with the most complicated thing. In Kha that's the C/C++ build system. It uses actual JavaScript code files for that, which is useful, but maybe we can do without that. What remains would be a file that contains include-dirs, #defines, and a list of patterns which define the files to compile. Kha doesn't currently natively support any kind of precompiled libraries. I strongly prefer source libs, because it's important to me that everybody who uses Kha can always step though all of the code. But on the other hand, we should probably just support both, including a lib isn't that complicated after all. Do other frameworks have any mechanisms for supporting C/C++ source code packages or is it all ndlls?

PS: I'm ok with JSON. I like Mr Crockford.

--
To post to this group haxe...@googlegroups.com
http://groups.google.com/group/haxelang?hl=en
---
You received this message because you are subscribed to the Google Groups "Haxe" group.
For more options, visit https://groups.google.com/d/optout.

[Nicolas Cannasse]

> To be acceptable to all frameworks, I think the config needs to be
> parsable by haxe code without a library (or agree on library, which has
> proved problematic) - which gives xml, json or some simple line-based
> ascii. I'm leaning towards json, or xml if we allow the "if" style
> conditionals. In the json case, I suggest multiple files like "common"
> "android" and "android-x86" which get ready sequentially and join
> together - but again open to ideas.

That's a very good thing to get started on this, I hope all the
framework authors will join this discussion and contribute.

Just one small suggestion regarding format. I'm fine with JSON, it is
much more easily to manipulate than XML. Regarding conditionals, we
could either have a generic { _if : cond, _then : a, [_else : b] }
format. Another option would be to run the JSON within hscript, which
would allow the following : { myconfig : if( cond ) "a" else "b" }

But consider it just an implementation detail to be discussed later, I
would not like to discussion to branch into technical details since we
need first to comeup with a more general agreement about doing something
regarding what Hugh listed.

Regarding Heaps : atm I don't have much config being required, I'm
trying to stick to using the haxe compiler "as it" without an extra tool
for compiling. I will need to differentiate between mobile and desktop
when I add support for it, as well as specify a few options for desktop
(window size for instance). I'ld gradly reuse a spec if the community
comes up with one.

Best,
Nicolas

[Justin Donaldson]

Good to see this discussion happening.  I'm mainly interested in the tooling support discussion that falls out of this.  If frameworks would just use some of the same terminology for describing details of the workflow, it would be a lot easier to support from an editor perspective.

[Rui Campos]

This a great idea, I'm very happy that this is finally being talked about. 

I'll just go through some random things that come to mind right now.

4. native hooks (eg, call something on startup/shutdown)

This would ideally be solved if we had an interface that all frameworks share, but the implementation differs per framework. Something like hooks for onStart, onShutdown, enterBackground, enterForeground. This interface should probably exist in c++/java/haxe code. (maybe part of the Haxe Std?)

Other important hooks that come to mind are the onRender and being able to queue stuff on the main haxe thread (this is very important for Android, as stuff comes on the UIThread, but haxe is usually running mainly on the GLThread. Maybe we can finally have a proper shared runloop standard interface that is accessible from not just haxe, but also Java and C++?

About JNI:

In the duell tool we have actually extracted the Haxe<->Java interface into a library so it's perfectly doable and I think it can perfectly live as a simple library and one of the first ones to be done with the new standard. https://github.com/gameduell/hxjni

JSON:

I don't really care too much about the format, we just picked xml because of the if's and because it can a lot more readable than JSON. Bottom line, JSON is more parsable, and XML is more readable. I don't think any choice is a bad one.

Keep the discussion rolling guys :)

[Hugh]

I think we all agree the file format does not matter too much - as long as we have a "reference implementation" that can read it.  If we include the JNI code and hook interface in this, we can have a "hxframework" library we can all share - as long as we all commit it this.

Just to clarify the hooks, I think "run on main thread" and "run on ui thread" would be appropriate functions - not 100% sure about onRender - this seems like a more framework specific thing.

Also the config - things like screen resolution I think can be left up to the frameworks specific config files - or lack thereof.  I'm not really taking about being able to share projects between frameworks, just libraries.

For small extensions (mainly just some glue code) I think we should find a source-code only solution - this will help mitigate the required-binary explosion you get on iOS.  In the case of ios, we can often just use haxe meta to do the config with @:buildXml, so you already have a kind-of haxelib only solution.  But for bigger extensions, or more complicated cross-platform libraries we need a more complex solution.

For me, the obvious solution is the existing cpp build.xml files.  There are already a lot of small and big libraries that are built with this and since the compile flags are shared, you get compatibility with the haxe generated cpp code.  To be clear, this is only for .cpp (.mm) code, not java or c# libraries or source code, assets or other config.  And of course, I could be biased, so correct me if you have other ideas.  Currently these are used with pre-build ndlls, git submodules, nme-dev style precompiled binaries and duelltool managed repos (not sure about Kha makefiles), so we can build on what is already there.  But I think what is missing here is a workflow.

That is, the developer installs a haxelib (or clones a git repo) which comes with source, but not binaries.  Ultimately they are going to want the compile binaries linked into their application, using the architecture(s) of choice.  There are a few ways of going here:

1. compile into a library in the directory relative the haxelib source code, using the existing architecture naming conventions

2. recompile into objs in the project's temp directory for every project

3. configure some kind of "staging area" that can be configured per-machine or per-project to store the pre-compiled object files.

4. combine 2 and 3 by using the existing hxcpp_compile_cache code - although I'm not sure how robust this is, so this may need improving.

1 I think is easiest, but putting obj files in a source code directory is not that great a solution.

2 Could be slow - especially for bit libraries used in multiple projects

4 Needs some work, but fixing that may also help generic hxcpp code (although not so much now with DCE)

Which leaves 3 as my favourite.  What I'm thinking is a hxframework function like:

useCacheDir = Framework.getDefaultCacheDir

// Tool does not care how this is generated - maybe it uses magic

pathToLibrary = Framework.findOrCreateLibrary(userCacheDir, "/path/to/extension", "architecture", ["debugflags?"] )

not sure about debug flags - maybe bool, or maybe arbitrary defines.

I think most libraries could be converted to work this way.  I can also change hxcpp to write the name of the library it generates or something like that.

I would like to convert hxcpp libraries over to use this, and avoid binary distributions in hxcpp - that would be sweet.  Maybe also move the getDefaultCacheDir and findOrCreateLibrary into the "hxcpp.Framework" class so I can keep these functions in sync with the hxcpp code - open to ideas.

How does this all sound?   We need 100% buy-in - if there are any hold-outs please speak up otherwise we may end up with yet-another-project-solution (yaps) rather than the one-project-system-to-rule-them-all.

Hugh

[Hugh]

I think it is worth looking at some specific too, for example https://github.com/haxenme/nme/blob/master/lib/NmeLink.xml

The result of this is a set of link flags, given the target architecture and also the compiler (HXCPP_MINGW). This is the "native format" for hxcpp linker.

So the question is, does anyone use anything other than the hxcpp compiler chain specification to build exes (.so) ? ie, your own logic for choosing a compiler exe and build flags?  If so, how does this logic go?

If not, I think the link api is almost already written, and could be something like;

hxcpplink path/to/lib1/buildfile.xml path/to/lib2/buildfile.xml -cacheDir my/special/project/cache

which in turn can be run from "haxe -D cacheDir=my/special/project/cache", and a @:buildXml injection - or maybe something similar but different.

Also for example, the hxcpp stdlib/regex/mysql/zlib libraries could easily be converted to this format, and the @:buildXml could be used to pull these in, build them once if required and link them in.

I think this can take care of the "static linking a c++/mm exe or android .so file" part of the config.

There are still other questions, like .ndll files.  Do these ship with the haxelib?  They probably should, since if you use neko, you might not have the c++ compiler installed.  There is also the ".dll" files that may or may not ship with haxelib - eg, LGPL code that requires a dll.  But if this in only needed when you know the developer has correctly configured compilers, maybe you want to build this on the developers machine.
Maybe the framework tool allows .ndll files to be build from the build.xml file in a standard way - this is pretty much already the case.

Well, that is some ideas about static linking - I'm pretty sure most frameworks work almost this way - with the possible exception of Kha, which I'm not sure of the final link step.

Again, this in only part of the specification - says nothing about java, c#, assets, manifests etc etc.

Hugh

[tondy]

my thoughts...

yes and no :)

yes - the project name,width,height, fps etc are good to be standardised

no - build process. too many targets, platforms, solutions etc

i'm not ready yet, but soon i'll put on github working snake game as example for "my solution".

with few words: i use bash-like hscript file make.hxs with config.json for building(embed resources) and the same config.json macro-converted

as Config.hx for the game itself. the rest of build options reside in the script file (make.hxs)

for desktop gui i use my own sdl-based ndll compiled with cmake - 1. is natural way, 2. i don't know how to do it with hxcpp

for android & apple i use openfl at the moment, but I plan to use their natural way for gui/audio - pure haxe/java for android and haxe/c++/objc for apple

p.s. +10 gradle, +10 json

[Hugh]

So I'm only talking about libraries here - feel free to start a new thread about "the project name,width,height, fps, embedded resources etc".  I have already stated that everyone's solution here is the best, and everyones else's is crap.

Here is an example of how to use hxcpp buildtool with sdl: https://github.com/haxenme/nme-dev/blob/master/project/buildfiles/sdl2.xml

There are other examples which are more inline-haxe based, which may suit you better by the sound of it.

From the library point of view, you have a "sdl-based ndll".  This is not a static library, so the library concept is a bit simpler - we need to work out a way of saying "please include this ndll in your project" - this is what I'm looking to standardize.

But  it also sounds to me like this is another framework, so in fact you would be a user of the framework library, not a target for it.

In this case, I'm looking for a way for you to get, for example, a free "facebook integration library" to work with your framework by perhaps supplying some kind of "hooks interface".  It is this library sharing I'm trying to facilitate here.

Hugh

[Rui Campos]

For simplification, I think we should go for a purely source approach. Meaning that we don't package any binaries, and just leave it to the framework to properly compile the extension. This is nice for compatibility, since people might have very specific build workflows or environments. The framework just makes an importer for these libraries converting the configuration to one that makes sense in the framework and then compiles it. 

We could start with a simple architecture, purely data driven, and see how hard it is to make it work on other frameworks. We could try for example a facebook library.

What I think we need to get started is:
 - A generic configuration file format that includes, e.g.:

   - asset folders

   - haxe compilation arguments

   - including other configuration files

   - if/unless

   - ... and more as they come

 - An extension of the configuration file for target specific configurations, e.g.:

   - native library configuration file linking (maybe just link to hxcpp build file for now?)

   - manifest lines

   - info.plist lines

   - frameworks

   - ... this can really explode, so we should be simplistic in the beginning and add more as we go.

 - An interface for all languages (c++, haxe, Java for now) that frameworks should implement, that includes. e.g.:

   - loadFile method

   - getDir, a version for cache, temp and static, where cache is permanent, temp is per session and static is read only. This should be cross platform enough

   - lifecycle callbacks (onStart, onExit, onBackground, onForeground, onLoop)

   - each platform can have platform specific functions, e.g. iOS can have push notification callbacks.

   - ... and more as they go.

 

 - Some libraries, that are implemented with the above described format, and make the bridging easier: 

   - Easy Haxe<->Java connection, the before mentioned JNI library.

The only big problem I see here is deciding on the format for how to compile the c++ code. Not everyone uses hxcpp for that, so it won't work for them.

Rui.

[Hugh]

This is pretty close to what I have in mind.

I have been thinking and discussing  more about the c++ native build, and actually it can be done today with the existing hxcpp.

That is, you can compile and link in any library for any hxcpp supported platform using the existing @:buildXml and extraparams.xml mechanisms - you do not even need a convention, since you can bootstrap it from haxe code.

What is missing is the hxcpp_compile_cache - this is there, but it is not properly tested.  So instead of reinventing anything, I just need to put the energy into getting this compiler cache to work 100%.

This will give haxelib developers an easy, frictionless option for static linking without distributing libraries.  For ios and android, which do not use ndlls, this is probably all you need.  I think libraries should still distribute ndlls since neko devs need not have c compilers installed.

Hugh 

[Marc Weber]

> For simplification, I think we should go for a purely source approach.

Thus start with mingw to get sqlite?

Then you need a full blown (cross platform?) package manager...

The best system allows both: binary and source distribution. nixos.org
comes close to what could be "perfect". But then you have both a package
management system and haxe library tools ..

I want to say "from source" is still not very accurate unless you talk
about deep dependencies (such as compiling sqlite or using binaries for
that).

Marc Weber

[Rui Campos]

Maybe my "purely" was misplaced, I meant code that is specific to the library. 

You should still be able to package external binaries. The Sqllite binary should be packaged in with the library that requires it, as well as other frameworks. But the code that is written specifically to interface with Haxe should be provided in source.

Rui

[Joshua Granick]

ANDROID

I have built an Android architecture that uses "Android library projects" because this enables the tools to merge Android manifest files, include needed interfaces, assets or other elements in an Android way, and is pretty close to the regular Android development style.

I would not be in favor of an architecture that requires new tags or special names for almost any Android manifest permutation. This is very difficult to merge and balloons quickly into something that only works under certain conditions

Lime uses an "extension-api" library project that extensions can reference, it uses a standard extension API in Java to expose almost anything that an extension would need: https://github.com/openfl/lime/blob/master/dependencies/extension-api/src/org/haxe/extension/Extension.java

It's up to the main activity class to call these hooks for extensions

IOS

I would like to see something similar for iOS, ideally if there is a format that allows for opening an extension in Xcode, defining custom plist values and having this merge into the project somehow. I'm not sure what mechanism is ideal for this?

OTHER

No matter platform, you need the ability to use custom built C++ libraries, prebuilt shared libraries, define Haxe custom flags, or haxelib dependencies, or perform certain kinds of logic. The "include.xml" approach, with its features, (I believe) has provided the flexibility needed to accomplish this

BINARIES

OpenFL and Lime support a "rebuild" command, which helps with extensions and other native libraries:

    lime rebuild my-extension-haxelib ios

    lime rebuild path/to/my/extension android

    lime rebuild path/to/my/extension/Build.xml windows

It looks for a "project/Build.xml" file, and passes standard defines and architectures in. If the library has no Build.xml, or has defines in the Build.xml so there is no build target for HXCPP to find, the build will be skipped. Certainly someone could do a custom build, but in most cases I find this more convenient.

IMHO, including binaries is more convenient for users, it's just a matter of having a build server, we have been doing this for Lime and a number of native extensions already :)

EXAMPLES

You can find examples here: https://github.com/openfl?query=extension

--

[Hugh]

I think there is only 1 reason for static pre-built libraries, and that is build-time performance.  If this can be solved, then there is no good reason I can think of.

There are a few more reasons for dynamic pre-built libraries - mainly build-time performance, neko, and reduced dependencies.  But this can't be done on iOS, so the point is moot for this platform.

I guess we can include the possibility of static libraries in the spec - but what I'm really hoping for is to make the haxe-based linking so compelling that you would not consider anything else.

I agree that the android library apk is a neat solution, but I have the feeling we should be moving to gradle, which presumably has similar mechanisms.  The other question here is whether the haxe.Template code needs to be used in conjunction with library, or whether the library mechanism  has enough flexibility on its own.

We also need some kind of convention for registering the extension.  For haxe/c++ code, you could probably use a static initializer function, although the timing of this may be wrong.  If the extension needs the "onCreate" callback, it is going to have to be registered as one of the first things the app does.  I think interfaces would give more flexibility to the implementation, rather than static-members.  I'm thinking something along the lines of a config that has:

registerExtension = "my.ext.ExtensionHandler", with "ExtensionHandler" a java class implementing the IExtension interface (onCreate etc), and the framework is required to call:

new my.ext.ExtensionHandle(this), where "this" implements the "IFramework" interface, which as "getAssetHandler", "getMainView" etc.

This is not too far off what you have already.

It would be worth looking over the existing extensions to see if we can exactly limit it to info.plist injection (and whether we need haxe.Template) and link frameworks or whether anything else is needed.  I guess the same extension callback system - but this time in c++ (mm?) rather than java.

On iOS, it is more practical to handle some of these callbacks in haxe directly because the threads and jni are not issues, so maybe the extension interface need not be so extensive.

Hugh

[Robert Konrad]

I have to admit, this whole discussion is a little difficult to follow for me. Kha's build system is very much completely different from what hxcpp normally does. It already creates gradle files for Android for example but never pulls in information on that level from a library (and I think it would be cool if we could avoid that or at least make it optional). A hxcpp_compile_cache is irrelevant for Kha, it already takes care of such things itself.

How about setting up a small sample project and continue the discussion based on that?

[Hugh]

I bit more on this topic - I think what Sven has done  (http://snowkit.org/2015/08/24/announcing-linc/) by essentially showing that something as complex as the whole SDL library can be be compiled in with the haxe code shows that this technique is extremely viable for any library that is used with hxcpp (ie, anything on iOS or android).  So think this is probably the best way to handle foreign source cpp code.

[Robert Konrad]

Hm, well, Kha always compiled everything in, so no objections here.

[Aymeric]

Hey, what about a repository shared between frameworks for Native Extensions? I'm refering to this post https://groups.google.com/forum/?hl=en#!topic/haxelang/QkQQ-PmCfQk