Haxe shared libraries, externs

[Jeff Ward]

What's the typical way to handle shared libraries where the source for both the library and the consumers are Haxe?

I can think of a few possible solutions:

• Write by hand both the implementations and the externs (analogous to c++ source and header files).
• Use some macro to generate Haxe extern classes from Haxe implementation classes.
• Use some compiler switch to tell the compiler -- this is an external class, don't include the implementation.

Do any of these automated solutions exist?

FYI, I'm targeting JS and AMD, so my plan is to require() the libraries and attach the implementations at runtime.

Thanks,

-Jeff

[Hugh]

Cppia does something like your third options.

At "host" generation time (in your case, "dll" generation), it outputs a list of classes that are included.  You could use an "onGenerate" macro for this.

When compiling the rest, it reads this output file and excludes classes already in the output.

You will also need some DCE solution.

https://github.com/HaxeFoundation/haxe/blob/development/std/cpp/cppia/HostClasses.hx

Hugh

[Mark Knol]

Maybe you could take a look at this haxelib: 

https://github.com/explorigin/modular-js

[Philippe Elsass]

Another approach:
https://github.com/elsassph/modular-haxe-example

However it would be improved if we find a way to actually generate externs. Either using a macro or the compiler's XML output.

This something I want to investigate myself to avoid recompiling complex/large dependencies.

Philippe

On 12 May 2016 09:38, "Mark Knol" <mark...@gmail.com> wrote:

Maybe you could take a look at this haxelib: 

https://github.com/explorigin/modular-js

--
To post to this group haxe...@googlegroups.com
http://groups.google.com/group/haxelang?hl=en
---
You received this message because you are subscribed to the Google Groups "Haxe" group.
For more options, visit https://groups.google.com/d/optout.

[Juraj Kirchheim]

However it would be improved if we find a way to actually generate externs. Either using a macro or the compiler's XML output.

The `--gen-hx-classes` compiler switch is your friend. Output is in hxclasses ;)

This something I want to investigate myself to avoid recompiling complex/large dependencies.

Experience has taught me that splitting your build into multiple JS files is impractical. If you pre-build a haxe/js library you lose inlining (and the optimization that comes with it, that potentially even avoids allocation as here: http://try.haxe.org/#0e9F9) and you can't DCE your code. You also can't call macros (although this could probably be solved by including them in the externs). You can't set compiler flags anymore. You need to be careful into which module you compile the standard-lib and (HxOverrides and all that jazz). You need to be careful to load that first. If you change a Haxe file, you need to know which module(s) to recompile. You need to make absolutely sure that your server's caching headers are properly configured (and that's assuming that the user's browser will handle them correctly), otherwise users might wind up having an outdated version of the core from cache and load a module from the server that is incompatible with it. Building is hard. Deployment is hard. Minification is hard (the more stuff gets renamed the higher the risk of losing compatibility). Total file size is much bigger.

I prefer to just configure my build with as many libs as I want and have enough discipline to keep the build compiler cache friendly (avoid circular dependencies between implementations, by writing them against interfaces ... not only does that make for better code, but also it restrains cache invalidation) and compatible with -dce full and ideally also closure's advanced mode. It's easier to maintain, faster to make a full build and also to rebuild (from cache that is), easier to deploy, faster to download and exhibits better runtime performance. It wins on any metric that developers or users would actually care about. Adhering to some quasi-religious best practice - "established" in another language (it's not like there are no JavaScript devs that wouldn't tell you to do the exact opposite, namely to bundle your files) that is fundamentally different from Haxe in almost every aspect beyond superficial similarity - is pointless. It's procrastination at best and waste of time and brain power at worst. You're going to be much better off if you just don't do it.

Sure, there are exceptions to every rule. But unless you have some well-isolated library that doesn't change all too often (in which case building it is pretty straight forward), I really don't think you want to prebuild it. It'll just lead to angry rants about how Haxe fights you at every turn, when in truth the fighting is initiated in the opposite direction ;)

Best,

Juraj

[Jeff Ward]

Thanks for the suggestion, Mark. My modules aren't arranged in a single build, so I don't think it's quite what I'm looking for.

> Another approach: https://github.com/elsassph/modular-haxe-example

Ah, fantastic Philippe, this looks like an example of what I want to do. I'll also have a look at what --gen-hx-classes does.

> It'll just lead to angry rants about how Haxe fights you at every turn, when in truth the fighting is initiated in the opposite direction ;)

Haha, sharp memory, Juraj. Let's be fair, I may be critical and angry, but at least I honestly want things to improve. :)

Hmm, you raise a lot of good considerations. But, as with many systems in computer science, you can't always avoid modularity and go for a monolithic build. For us, we want well-defined, reusable components, they're built individually, and not all of our components are Haxe sourced (we'll have support for typescript, plain js, etc and support for a variety of component loaders: polymer, riotjs, etc.)

We have our server-side considerations taken care of, with content-hash filenames, with the complete manifest definition delivered on the page, so you'll get one whole version or another, no mix-ups. We declare our dependencies and (using requirejs) ensure they're loaded in the right order.

Still, losing DCE and inlining isn't ideal. It does make me consider -- we could potentially setup a monolithic build around all our Haxe-based components, then require() at runtime the monolithic library for any component that's Haxe-based. It'd have the downside that using any of these components on any pages would load the whole Haxe-based library, but perhaps that's not a big deal, assuming users will cache it the first time.

Alright, well, there're certainly interesting things to investigate here, thanks guys!

Best,

-Jeff

[Philippe Elsass]

Juraj, thanks for --gen-hx-classes.

And thanks for reminding the caveats of not doing a single build, but that said, please accept that others may still want to do it in some specific scenarios. 

This quasi-religious tendency in the Haxe community to only consider monolithic binaries isn't helpful and respectful of other people's intelligence and use cases...

Jeff, you seem to be working on a rather complex system! I think you could still bundle your Haxe components separately and use the modular approach to share/reuse the common code.

Philippe

--

To post to this group haxe...@googlegroups.com
http://groups.google.com/group/haxelang?hl=en
---
You received this message because you are subscribed to the Google Groups "Haxe" group.
For more options, visit https://groups.google.com/d/optout.

--

Philippe

[Dion Whitehead Amago]

With all due respect to others needing to find their own solutions, my experience in putting Haxe node.js/client javascript apps in production mirrors Juraj's. 

Splitting out Haxe compiled code into multiple modules then integrating them later massively increases your surface area for complexity, mistakes, bugs, and overall devops and team management/project work.

I would make sure you *really* need to do it.

[我每天睡不着]

// haxe dev version
// 
#if b_js extern #end class A {
    public function new():Void {
        trace("new A");
    }
}

#if a_js extern #end class B {
    public function new():Void {
        trace("new B");
    }
}

[Dion Whitehead Amago]

Maybe I'm missing something, but can't you just use something like git submodules?

[Jeff Ward]

我每天睡不着 - that's clever. However, you start running into problems when, e.g., initializing static vars:

#if b_js extern #end class A {

    public static var foo:String = "abc";
    public function new():Void {

This gives the error: Extern non-inline variables may not be initialized

This does seem a little odd. If the compiler ignores a function implementation in an extern class, why wouldn't it ignore an initializer to a static member?

Anway, you could start if-def'ing out the init values, but then it becomes quite tedious.

Another is relying on inference for function parameter types:

    public function foo(param=false):Void {

This gives the error: Type required for extern classes and interfaces

So, implemenation code isn't quite compatible with extern definitions as one would hope. But thanks for the idea! It does work if the definition is extern compatible.

Dion -- while these modules work fine if all classes are compiled into each module, but that's a waste of filesize. The whole goal is to make sure class implementations aren't duplicated in each module.js file. For bonus points, it should apply to Haxe std library classes as well. I don't see how git helps with that.

Best,

-Jeff

[我每天睡不着]

error: Type required for extern classes and interfaces

for extern class, every field/variable/params need explicitly type

#if a_js extern #end class B {

    static var value:String;
    public static function main():Void{
    }
    public function foo(param:Bool=false):Void {
    }
#if !a_js
    static function __init__(){
        value = "foooooooo";
    }
#end
}

[Jeff Ward]

Thanks again, 我每天睡不着, but I don't want to have to structure my code in a specific way to achieve this.

Indeed, --gen-hx-classes does exactly what I originally asked: it spits out extern difinitions for all of the classes in the compile. Excellent!

Combining --gen-hx-classes with Philippe's example (the Stub.hx macro that alters the generated output), I am able to get exactly what I want (which is not necessarily what everyone else wants -- there are many ways to achieve to modularity):

The major decision that I made that others might not necessarily is to compile one shared library compiled against all components. This way I retain DCE (compiling all haxe standard classes to JS would be impractical for a website.) Thus I get all (and only) the haxe standard classes I use (e.g. Std and Array and Math and haxe.crypto.Base64 and haxe.crypto.Bytes, etc) as part of the lib. Then I add an @:native() to each extern class (so the apps look for $hxshared.haxe.crypto.Base64 instead of the global haxe.crypto.Base64. Then I compile each component against the generated externs, and wrap it in a require() call to get the $hxshared library at runtime.

So it's not a traditional modular system (compile the bits separately), but since we "deploy a website", there is a single build time at which all this happens, and I get runtime modular pieces without duplicated code. Good stuff!

One thing I noticed -- the compiler still generates extern class definitions for classes that are --macro excluded, which is a little annoying, I must delete those before compiling the apps.

I'll post an example later if I get it together -- there are still a few things hardcoded, but it conveys the idea.

[Jeff Ward]

As I think about it, the decision to have a single global library is unnecessary -- you could run this "monolithic" step as many times as you like to generate libraries. Each lib would need to declare its contents & dependencies, compiling each lib against all apps and the generated externs of its dependencies.

It would help if extern classes weren't generated for excluded classes. Maybe I'll file an issue with the compiler... Someone probably knows why it's done this way, then my feature request will be debated ad nauseam, and finally scheduled for Haxe 6.2. :D ;)

But in my world, I'd still use monolithic lib builds to retain DCE and deploy as little code as possible.

[Jeff Ward]

Questions:

1. Does the compiler or any other tool provide the answer to "What types are defined in a given class path?" My cheap solution is rgrep .hx files for class / enum / (abstract?) definitions.
   
2. Is there a macro I can invoke from the --macro commandline that does exactly what @:native('') does on an extern? Because I need to rescope library classes, but I can't compile against externs.

Status:

Ok, so it turns out you can't directly build against generated hxclasses - for one thing it reports static inline var as:

static var ELEMENT_NODE(inline,never) : Int;

Which is invalid syntax. But how is it supposed to handle inline definitions. Clearly it cannot. But this is another reason the monolithic build looks good -- it solves both the DCE and the inline problems.

So instead of -cp hxclasses, I grep for class/enum definitions in hxclasses, and excluded them from later builds.

So while I can't post a code example (this is being built into our proprietary build system), here're the steps I've got so far. This is still a WIP:

1. Figure out your build order via dependencies or whatever and build the lowest (global / catch-all) library first
2. Repeat these build steps from the bottom up:
1. Exclude all types you want in all other components.
1. e.g. if you want a component to include haxe.crypto or promhx, exclude them from the global build: --macro 'exclude("haxe.crypto")'
2. Exclude all types you've already built, and rename them (@:native)
3. Build with -D js-classic (we'll wrap it later)
   
4. Grep the hxclasses for class names to exclude from later builds, and keep track of which lib they come from (for renaming)
3. For the Main file for the global build (and any library builds without a Main), generate a Main class that references (not calls) the constructor of every component with a constructor. Discard these DummyMain.main() functions in the post processing step.
4. Post-processing js (ala Philippe's Stub.hx):
1. Remove DummyMain.main definition and call from library builds
2. this-scope all top-level vars (lines that start with var, no spaces):
   var haxe_io_Bytes = this.haxe.io.Bytes = function...
   
3. Wrap in an AMD define(), name appropriately, and return the this Object

Best,

-Jeff

[Jeff Ward]

On Tuesday, May 17, 2016 at 7:59:01 AM UTC-6, Jeff Ward wrote:

 

Is there a macro I can invoke from the --macro commandline that does exactly what @:native('') does on an extern? Because I need to rescope library classes, but I can't compile against externs.

Ah, excellent:

  --macro addMetadata('@:native("$mylib1.haxe.crypto.Base64")', 'haxe.crypto.Base64')

So this causes the consumers to look for the Base64 lib under that namespace, and you setup your define/requires like so:

define('myapp', ['mylib1'], function($mylib1) {

...

});

And recall your post-processed mylib1.js lib setup, something like:

define('mylib1', [], function() {

  ...

  var haxe_crypto_Base64 = this.haxe.crypto.Base64 = function ...

  ...

  return this;

});