Tool to convert javascript to Haxe Extern class files

Showing 1-20 of 20 messages
Tool to convert javascript to Haxe Extern class files Huck 5/15/12 6:18 PM
Hello Haxers,

Im here to propose a solution to one of the biggest shortcommings of haxe on the javascript front. Haxe is really cool, but the fact that it is not compatible with javascript is a major issue to its success. Sure, we can create extern classes, but its a REAL pain to create and worst, they fall back behind in versions behind the javascript ones becuase they requuier so muhc maintenance. For any javascript developer, this is a major turnoff, and seriously, most people dont want to make their own externs for HUGE javascript libraries, they just want it to work.

so, i am here to offer a solution to this problem. I need to develop a website in haxe but needed to use jquery, jquery-ui and three.js. the haxe libraries for these files if they exist are ususally poorly maintained and out of date. so, i thought it would be cool to create a tool to convert javascript into a "best shot" version of haxe extern files. With a tool like this, it would become almost trivial to generate plenty of extern files for any javascript out there. The tool i created here is a demo, a hack i made to test the concept. It takes in a javascript file and generates a haxe extern file for all classes in the file. sure javascript doesnt have type info, but the tool can try to guess the types from the code and provides a best guess. the rest is set as Dynamic. So the tool greatly simplifies the operation of creating the externs, and then all is required is a quick sweep by a human to do the last touches. with a tool like this, if it is properly developped can greatly help haxe become the standard on the web.

so about the tool, it is written in c# (runs in mono perfectly too so on all plateforms) it uses JInt library to parse the javascript files (which uses ANTLR3). this library is highly efficient and parses successfully the most difficult javascript files. i tested against three.js and jquery.js. so this library parses the JS and returns a DOM of the code, which we can analyse and use to generate haxe code.

now, the project i made here is a demo and im ashamed to say the code is seriously crappy. it mostly works, but is far from perfect in current state. thats mostly because i dont have time to dedicate to this, so it was a hackfest for me in a single day to put this together to prove the concept. If i was to take put more time into it, i would rebuild it but with a proper structure and it could be very good. javascriopt is fairly simple, but has its quirks. So now, i suggest a haxe developper put just a few days into this tool to recreate it proper and it can become extremely powerful, and really give a big push to haxe on the web with javascript. As i said, if you look at the code you will see it is very easy to parse the javascript and recreate the structure for haxe.

So, again i hop someone will take this code and build it as it should, because if this tool works, it wll be a HUGE bonus for haxe, one that google dart doesnt even have yet... im here if you have questions,

regards,

note: i provide this code as is, with no warranty. i also dont have any time to put into this, so its a donation (hacky code anyways ;)).

note 2: im having troubles uploading the code, so i posted the code on this URL until i can get it to work on google groups.
 
Re: Tool to convert javascript to Haxe Extern class files Huck 5/15/12 6:19 PM
there we go, worked this time :)
Re: [haxe] Tool to convert javascript to Haxe Extern class files FullOfCaffeine 5/15/12 6:34 PM
I also think Haxe for js dev can be the best thing since sliced bread. This sounds interesting, why not pushing to github? :)

I totally agree that the maintenance of externs is a pain.

As a side note though, I think the best thing would be to somehow include that into the Haxe compiler itself, as part of the js-gen stuff? Or, create a build tool and compile to Neko.

My two cents,

- Marcelo


unk...@googlegroups.com 5/15/12 6:35 PM <This message has been deleted.>
Re: Tool to convert javascript to Haxe Extern class files Huck 5/15/12 7:03 PM

And here is a sample haxe file generated by the tool for Three.js version r49. Sure its not perfect, but its pretty good considering it was created by a single day hackathon and that the tool generated this file in a single second.

Again if another dev gives it a few days of proper love, REALLY good stuff can come out of this idea for haxe.

regards,


Re: [haxe] Re: Tool to convert javascript to Haxe Extern class files Philippe Elsass 5/16/12 12:45 AM
Looks like a good direction, three.js is a particularly complex source...

Another option, with JS, is to load the library in a browser and to explore it "live" - but the parser solution gives you a chance to get the comments.




--
Philippe
Re: [haxe] Re: Tool to convert javascript to Haxe Extern class files Tony Polinelli 5/16/12 2:52 AM
nice idea- it seems to produce a lot of Dynamic's tho - wouldnt this
make the usefullness of haxe  a little limited ?
Tony Polinelli
http://touchmypixel.com
Re: [haxe] Tool to convert javascript to Haxe Extern class files Nicolas Cannasse 5/16/12 3:52 AM
Le 16/05/2012 03:18, Huck a �crit :
> Hello Haxers,
>
> Im here to propose a solution to one of the biggest shortcommings of
> haxe on the javascript front. Haxe is really cool, but the fact that it
> is not compatible with javascript is a major issue to its success. Sure,
> we can create extern classes, but its a REAL pain to create and worst,
> they fall back behind in versions behind the javascript ones becuase
> they requuier so muhc maintenance. For any javascript developer, this is
> a major turnoff, and seriously, most people dont want to make their own
> externs for HUGE javascript libraries, they just want it to work.
[...]

Hi,

While I completely agree with the issue itself, I'm not sure an
automated tool is the best solution.

Haxe developers should expect that quality externs are available, and by
"quality" I mean that people took time to read the library documentation
and strictly type the extern API, add the overloads, etc.

I think there's several approachs for that :

- using an automatic tool to allow to extract the first "untyped"
version, then work on make it typed

- extract directly the typed version from the HTML documentation of the
library, but there is no real documentation standard and it can break
easily if they change the output/engine

For instance, I'm using an automated extracting tool for Flash API -
which is a lot more easy since it's strictly typed - but I still have to
do some extra manual work to add a bit more strict typing (using enums
instead of String where it makes sense) and other things such as
@:require version annotations.

In the end, I think what we need is people who have enough time to
dedicate maintaining good quality externs, I'm not sure we can do it
magically.

Anyway thank you for the effort, I'm sure it will help other people.
Might be quite complementary with what Joshua is doing.

Best,
Nicolas

Re: [haxe] Tool to convert javascript to Haxe Extern class files Huck 5/16/12 5:43 AM
Here are some clarifications

becasue of javascript's untyped nature, its impossible to determine ALL the parameter types. but, it is possible to determine many from the code itself (variable assignments, return values, etc.).The example i produced here was a quick prototype and yes it has a lot of dynamics, but i did only 5% of the deduction work,  if the tool was properly built, we could find much much more and get at least 80% of data types correctly filed in.

now @ nicholas: for sure any class generated by a tool would have to be reviewed by a human, but at least the tool can make 98% of the work, leaving humans with a simple review to add whats missing. It would become trivial to maintain and add new externs for the fast moving javascript world. Without it, all the work must be done manually, and i have enough experience in software programming and opensource work to know that versions fall behind, always. it is even the case now. three.js is at r47 and jquery is largely incomplete. and jquery-ui? and other proprietary javascript libraries?

now, sadly i dont have any more time to dedicate to this tool, im on short shcedule project, but i wanted to get the ball rolling and demonstrate the large potential it has. This tool will make a huge difference in haxe ecosystem, but for this it has to be developed. the code i posted here is only a prototype, but it needs a developer assigned a few days on it to start from scratch and redesign it correctly using Jint and build this tool.

i really hope someone can take the ball from here and work on it. and remember, even google dart doesn't have it and its a huge limiting factor for it as well. here is a chance to get ahead before coffeescript takes it all...
Re: [haxe] Tool to convert javascript to Haxe Extern class files clemos 5/16/12 6:13 AM
Hi

The tool looks very promising.

I have also been working on a similar tool for PHP, which takes
peardoc xmls and generates externs from them.
I also made one before for Java / javadoc, when I was working on HaXe
+ Red5 with Rhino...

I have never released any of them because there are still two quite
big issues I never really managed to solve:

- you'll likely need to manually change some definitions, to type them
more, change them or correct them, etc,
and then all this is deleted when you generate the externs again...
So you need a good "patching" mecanism to have enough flexibility.

- in PHP there is a quite annoying issue with (anonymous) objects vs
associative arrays.
Most PHP APIs will use associative array, so you have to convert these
to anonymous at runtime, and the other way round...
I think it may be possible to do this more or less cleanly with
macros, but I have never had the time to really try that.

Anyway, I have had a quick look at Joshua's BuildHx and it looks
really promising.
https://github.com/jgranick/buildhx
It seems to solve my first issue by providing a nice way to alter
generated definitions with a .xml "patch".
Moreover it seems modular and easy to extend. This was not the case
for my projects either...

I think we should join forces on this, and use a common structure to
solve the extern generation, and BuildHx is a very good start.
I'll try to add php support to it when I have some time.
I think you may also want to check this project and maybe contribute.

Regards,
Clément
Re: [haxe] Tool to convert javascript to Haxe Extern class files Huck 5/16/12 6:27 AM
cool, now were talking :)

the big "breakthrough" in my tool above is the use if Jint (http://jint.codeplex.com/) which does an amazing job at parsing even the most complicated javascript files (parsed jquery and three without a hintch) adn returns a DOM of the code, easy to read. This in itself is very powerful as it allows to reverse engineer the code, and generate "something". be it haxe externs or more.

now buildhx still seems to require JSDuck tags, and xml definitions, these must still be written and maintained, its one evil for another. So the use of Jint could be the perfect complement to buildhx. Instead of generating .hx externs with Jint, we could generate the xmls, and then merge the changes with existing ones. Then after use buildhx to compile them. The way we do it to me is irrelevant, as long as we do it.

i think that if we can fix the javascript/haxe dependence mismatch, haxe can truly shine and become mass market in the HUGE javascript world.

in the end, i agree we should all give some time to this issue, as it will bring more users, and thus a bigger and more alive community; good for us.
Re: [haxe] Tool to convert javascript to Haxe Extern class files Huck 5/16/12 6:35 AM
a quick addendum to aboce. The problem with buildhx is that it requires JSDuck annotations, but most javascript projects dont have them, and we cant expect that they will add them. so it kinda kills the purpose of the tool for a pan-javascript world generator.

now, an idea occurred to me. if i was to redesign this tool, i would proceed by parsing the javascript DOM created by Jint, and recreate one in memory that maps to "any" OOP language. (example, class A, has 2 variables, B and C, and ahh, this function determines B as an Int, and ahh, this one determines C as a class with this prototype...).  so we would end up with our own custom DOM in memory, and from this DOM, i could output haxe externs, or literally convert the javascript to full haxe, or c++, or java, etc. now the beauty of this is that this DOM in memory could be serialized to xml, yaml or whatever. the users could then add their refinements to the missing parts of this xml used to generate the JavaScript. and reloaded at every new generation. we could then add the delta from the javascript, while maintaining the changes done by hand. hence allowing to keep improvements.

just ideas here. and in another design approach it could be used to generate either JSDuck metadata for buildhx, or the xmls themselves.

lots of possibilities indeed from actually deep reading into the javascript of ANY library..
Re: [haxe] Tool to convert javascript to Haxe Extern class files clemos 5/16/12 6:44 AM
Hi Huck

From what I understand BuildHx "currently" uses JSDuck,
But one could create his own parser to support other input material
(see: https://github.com/jgranick/buildhx/tree/master/buildhx/parsers)
At least, that's what I was planning to do for PHP/peardoc...

Regards,
Clément
Re: [haxe] Tool to convert javascript to Haxe Extern class files Joshua Granick 5/16/12 10:19 AM
BuildHX was built off an older tool I created which was able to parse  
JSDuck output to generate JS extern classes automatically.

I recently improved on the prior tool to open the door for multiple kinds  
of parsing, as well as manual definitions in an XML file. When you are  
able to parse documents to generate the definitions automatically, you may  
still need to tweak or adjust the types by hand. BuildHX is designed to  
allow this workflow.

Fintan Boyle recently contributed a YUIDoc parser, so you can use BuildHX  
to generate externs for libraries like the EaselJS framework.

You can use BuildHX in cases where you do not have an automated solution  
to load the types and definitions for the target library. The "joapp" and  
"box2d-native" projects that I have on GitHub are both manually defined.

If you are able to automatically derive types using JInt, you could create  
a JIntParser for BuildHX, which could read a source directory and populate  
the types it finds there:


  <source path="my/js/files" parser="jint" />



Similar to what I have done for Sencha Touch, you could fill in the blanks  
manually with the rest of the XML definition file for any given library.

I am also investigating Doxygen, which can help export types for C++  
libraries. It would be great to have an automated path for generating C++  
bindings.

BuildHX is not written in C#... it's in Haxe, of course, but that makes it  
easier for it to run on multiple platforms using Neko.
Re: [haxe] Tool to convert javascript to Haxe Extern class files Cauê Waneck 5/16/12 10:26 AM
I recently improved on the prior tool to open the door for multiple kinds of parsing, as well as manual definitions in an XML file. When you are able to parse documents to generate the definitions automatically, you may still need to tweak or adjust the types by hand. BuildHX is designed to allow this workflow.

That's very cool! 

If you are able to automatically derive types using JInt, you could create a JIntParser for BuildHX, which could read a source directory and populate the types it finds there:

 <source path="my/js/files" parser="jint" />

Similar to what I have done for Sencha Touch, you could fill in the blanks manually with the rest of the XML definition file for any given library.
BuildHX is not written in C#... it's in Haxe, of course, but that makes it easier for it to run on multiple platforms using Neko.

I can try to compile it with the c# generator, It'll be a good test, and we'll be able to use Jint! :) 
Re: [haxe] Tool to convert javascript to Haxe Extern class files Huck 5/16/12 10:38 AM
Cool!  things are getting really interesting now :) I think using Jint to supplement buildhx is a great idea.

I have trouble finding free time, but at the same time i believe this is a really good cause, so let me know if i can help in any way (only can give so much time though).

for one i've had some experience yesterday with the output from Jint, so i can definitely give some advice on how to use it.

first, throw away the code i did yesterday, its a mess and should be considered for instructional purposes only.

but Jint provides a DOM of the javascript code which can be parsed, each line is a statement, and each statement is made of various expression clauses of different nature. So i suggest building an iterator for each statement/expression from the Jint DOM and build an internal representation of the useful data. each time a returnStatement or an assignment expression is found, use the Member info to lookup the types and restore type information from code analysis. then this internal DOm could be serialized both ways into XML, and/or merged with the buildhx xml format. every javascript parsing could apply the delta to the exising DOM, hence updating the definition with each successive new javascript file version.

might seem complicated but its actually pretty simple, the trick is just to build a proper iterative analyser for the Jint DOM. let me know if there is anything else i can do to help.

regards,
Re: [haxe] Tool to convert javascript to Haxe Extern class files Cauê Waneck 5/16/12 10:43 AM
Cool! I'll have a look at what you did later on! :)


Re: [haxe] Tool to convert javascript to Haxe Extern class files Joshua Granick 5/16/12 11:14 AM
Since BuildHX is compiled for Neko, it can be distributed and used from haxelib. Ideally, I would like to continue this setup, so I don't think the tool should be compiled as C#.

However, we could create a standalone Jint-based executable and put it in the "/bin" directory along with jsduck and (probably) doxygen. It could accept a directory of JS files, and output type information in JSON or XML files.

These could either be formatted natively for BuildHX, or BuildHX could have a parser to understand it. I would recommend the latter, since that would allow much more verbose type information (dumping almost all the information you can get) which we could then sort through and use. Although there is information that would not be used for now, you never know how things might grow in the future. Having a verbose output format may make it easier to improve later in BuildHX.
Re: [haxe] Tool to convert javascript to Haxe Extern class files Huck 5/16/12 11:17 AM
yeah, this is how i would see it too, as an external utility tool to be invoked from buildhx. thanks to mono it can run pretty much on any platform neko can.
Re: [haxe] Tool to convert javascript to Haxe Extern class files Cauê Waneck 5/16/12 11:28 AM
Since BuildHX is compiled for Neko, it can be distributed and used from haxelib. Ideally, I would like to continue this setup, so I don't think the tool should be compiled as C#.

Yes, makes sense. Even though the neko app.n could just invoke e.g. mono or the executable, it really means that we will have to depend on the user having mono or .Net runtime installed.

However, we could create a standalone Jint-based executable and put it in the "/bin" directory along with jsduck and (probably) doxygen. It could accept a directory of JS files, and output 
type information in JSON or XML files.

Ok, seems good! I'll leave it to use the C# target for an actual project later then ! hahaha