Any plans to get Julia to be fully self-hosting?

[Dzhelil Rufat]

As the maturity of a language is measured by whether it is implemented in itself, I think it is important to get the Julia language to be self-hosting.

My understanding is that the tokenizer and parser for julia are implemented in a dialect of scheme called femtolisp? Are there any plans to migrate the frontend wholly to Julia?

Additionally, for the backend, can one create julia bindings to llvm, and then use those to write the backend completely in julia as well? What are the  obstacles to getting Julia to be fully self-hosting that right now? It would be really great if we could one day have a julia language implemented completely in julia.

[Stefan Karpinski]

This would definitely be good, but it's a lot of work with no user-facing benefit, so it's a pretty low priority. If you're interested in working on it, however, it would certainly be nice. I kind of suspect that being self-hosting is mainly a measure of language maturity because only in fairly mature languages have all the things more important than being self-hosting already been taken care of.

[Jeff Bezanson]

I agree this would be cool, but it can't be the only measure of
language maturity --- not every language is designed for writing
compilers.

The main problem is that this involves lots of tedious work, after
which you end up with the same thing you started with, just
implemented differently. I'm not even sure I want to use LLVM in any
language but C++, since that's the language it's documented in and C++
APIs tend to be highly coupled to the language.

But, as our static compilation emerges and improves (thanks to
Jameson), and C++ interop emerges (thanks to Keno and Isaiah), this
may gradually become a reasonable thing to do. But we can't really
afford to prioritize being self-hosting for its own sake; the real
value is in these steps along the way (static compiler, C++ interop,
etc.)

On Sat, Nov 23, 2013 at 4:52 PM, Dzhelil Rufat <dzh...@gmail.com> wrote:

[Ivar Nesje]

Can someone explain to me what the benefit of self-hosting a language is?

Links would also be helpful.

[Stefan Karpinski]

The biggest benefit for us would be that there would be a one-to-one correspondence between data types in the parser language and the program being parsed. That means you can parse literals as their values and represent expressions with expression objects. For example the effort to support 128-bit integer literals is somewhat complicated by the lack of 128-bit integer types in flisp. If the parser was written in Julia, you would have a natural representation for every possible kind of literal.

[Kevin Squire]

It would also make hacking on the parser easier.  I actually like lisp, but I don't make any claims about being good at it, and at this time, julia is much easier to deal with.

Although, at this stage in language development, it might actually be better that only a few people are hacking on the parser. ;-)

Kevin

[Jiahao Chen]

On Saturday, November 23, 2013 4:52:44 PM UTC-5, Dzhelil Rufat wrote:

As the maturity of a language is measured by whether it is implemented in itself, I think it is important to get the Julia language to be self-hosting.

Counterexample: Fortran.

 

[Stefan Karpinski]

While being self-hosting is not essential, I do think it would be really convenient, so I don't want to discourage any work in that direction. Homoiconicity makes it particularly handy since the parser can literally just generate Expr objects and lowering passes can then work with those, much like our type inference code does already.

[Jeff Bezanson]

The parser already does generate Expr objects, via the C shim in
ast.c. And if we wanted to support more types of literals,
julia-parser.scm could return expressions like (int128 lo-bits
hi-bits), which would also be translated by this shim. Nobody ever
sees this intermediate stage except the shim, so while this is
slightly clumsy it's not a serious limitation.

[Stefan Karpinski]

Well, yes, if you include ast.c in "the parser", but I was considering only the Scheme code to be "the parser" proper.

[Gour]

On Sat, 23 Nov 2013 17:03:38 -0500
Jeff Bezanson <jeff.b...@gmail.com>
wrote:

> But, as our static compilation emerges and improves (thanks to
> Jameson), and C++ interop emerges (thanks to Keno and Isaiah), this
> may gradually become a reasonable thing to do.

What is the current status and/or estimation when Julia might provide
statically compiled executables?



Sincerely,
Gour

--
One who is not disturbed in mind even amidst the threefold
miseries or elated when there is happiness, and who is free
from attachment, fear and anger, is called a sage of steady mind.

http://www.atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

[Tobi]

Am Sonntag, 16. Februar 2014 21:55:17 UTC+1 schrieb Gour:

What is the current status and/or estimation when Julia might provide
statically compiled executables?

What is the main motivation to have binary executables? Using the C-API it should be quite simple to write a small C program that links to libjulia and executes some code.

[Gour]

On Sun, 16 Feb 2014 13:45:47 -0800 (PST)
Tobi <tobias...@googlemail.com> wrote:

> What is the main motivation to have binary executables? Using the
> C-API it should be quite simple to write a small C program that links
> to libjulia and executes some code.

I am thinking about statically compiled and linked application -
convenience is certainly one aspecti when one wants to deliver e.g.
complete GUI app, but I guess it by using type-annotations, one should
achieve even better performance?


Sincerely,
Gour

--
A self-realized man has no purpose to fulfill in the discharge
of his prescribed duties, nor has he any reason not to perform
such work. Nor has he any need to depend on any other living being.

[Stefan Karpinski]

This is not going to happen any time soon – there are a lot of fairly large steps to get to fully static compilation. It will, however, happen.

[Gour]

On Sun, 16 Feb 2014 18:23:25 -0500
Stefan Karpinski <ste...@karpinski.org> wrote:

> This is not going to happen any time soon – there are a lot of fairly
> large steps to get to fully static compilation.

Do you mean it's post-1.0 target?

> It will, however, happen.

Very nice.


Sincerely,
Gour

--
Even if you are considered to be the most sinful of all sinners,
when you are situated in the boat of transcendental knowledge
you will be able to cross over the ocean of miseries.

[Tobi]

Am Sonntag, 16. Februar 2014 23:00:50 UTC+1 schrieb Gour:

I am thinking about statically compiled and linked application -
convenience is certainly one aspecti when one wants to deliver e.g.
complete GUI app, but I guess it by using type-annotations, one should
achieve even better performance?

No, it won't be faster in general. How can it be as Julia is almost as fast as C?
Furthermore I have doubts that a statically compiled binary application is more convenient.
We certainly need an easy way to distribute Julia applications but static compilation is IMHO not the crutial thing to get there.
Any application I can think of will use some packages and the question is more how to get all these things together and install
it on a user PC.

That beeing said there currently is one performance bottleneck which is package loading time. For the base module this has been
solved by static compilation of the system image (sys.dll/dylib/so). For fast package loading time one can extend the system
image by including packages in it (This is very recent work!). But this is only a stopgap solution. At some point we can hopefully
precompile packages.

[Tobi]

P.S.: Would be also interesting what you are planing to do. Should your application be cross platform?

[Gour]

On Mon, 17 Feb 2014 00:54:58 -0800 (PST)
Tobi <tobias...@googlemail.com> wrote:

> P.S.: Would be also interesting what you are planing to do. Should
> your application be cross platform?

Yes, I'll develop on Free/PC-BSD, but would like to be able to provide
application for both Mac OS & Windows users...however, here there is a
question of GUI bindings, but that was already addressed here.


Moreover, yesterday I've found out that Qt bindings is one of the ideas
for Nimrod's GSoC projects, so I'll keep an eye on that language as
well.


Sincerely,
Gour

--
An intelligent person does not take part in the sources of misery,
which are due to contact with the material senses. O son of Kuntī,
such pleasures have a beginning and an end, and so the wise man does
not delight in them.

[Tobi]

If you want to make a distributable cross-platform app, you will have to make custom installers for Windows and
OSX. This is some effort but justified if the project is not to small. But the installer will have to handle not only
Julia (or Nimrod) but also the Qt libraries if you want to use that toolkit.

As your most important thing seems to be that you are using Qt, you could also use C++ for your GUI and use
embedded Julia for the computations. In the current state this seems to be the most robust solution.

[Gour]

On Mon, 17 Feb 2014 02:11:36 -0800 (PST)
Tobi <tobias...@googlemail.com> wrote:

> If you want to make a distributable cross-platform app, you will have
> to make custom installers for Windows and OSX. This is some effort but
> justified if the project is not to small.

I'm aware of it considering that app might be interesting for not so
tech savvy users and Mac/Windows users are accustomed to install and not
to build their apps.

> But the installer will have to handle not only Julia (or Nimrod) but
> also the Qt libraries if you want to use that toolkit.

Yes, but it's the same no matter which GUI toolkit one uses.

> As your most important thing seems to be that you are using Qt, you
> could also use C++ for your GUI and use embedded Julia for the
> computations. In the current state this seems to be the most robust
> solution.

I'd simply like to avoide using C(++) and focus on my problem using
higher-level language which provides enough speed.

As far as Qt is concerned, wx is also viable alternative, but Qt seems
to be the best option today for multi-platform app due to having much
more developing activity than wx and, as I wrote earlier, wherever one
goes, nobody recommends using GTK for anything except Linux...actually I
just hear the word 'pain' mentioned often in regard.


Sincerely,
Gour

--
Not by merely abstaining from work can one achieve freedom
from reaction, nor by renunciation alone can one attain perfection.

[Gour]

On Mon, 17 Feb 2014 00:53:04 -0800 (PST)
Tobi <tobias...@googlemail.com> wrote:

> No, it won't be faster in general. How can it be as Julia is almost
> as fast as C?

Well, afaik, it's still in the range of 2X.

> Furthermore I have doubts that a statically compiled binary
> application is more convenient.

I did not express clearly - I was not thinking about statically compiled
and linked application, but about the speed difference between using JIT
compiler and native code generation as it is e.g. between Julia and
Nimrod.

For other external libs, I do not mind using share libs avaialble on my
native OS.


Sincerely,
Gour

--
As the embodied soul continuously passes, in this body,
from boyhood to youth to old age, the soul similarly passes
into another body at death. A sober person is not bewildered
by such a change.

http://www.atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

--
The senses, the mind and the intelligence are the sitting places
of this lust. Through them lust covers the real knowledge of the
living entity and bewilders him.

[Tobi]

Am Montag, 17. Februar 2014 11:27:43 UTC+1 schrieb Gour:

I'd simply like to avoide using C(++) and focus on my problem using
higher-level language which provides enough speed.

You will always have to make compromises. The holy grail is not there yet.
If your project starts now, you have to make a decision. And if you want to use
Julia the options are:

- Qt(C++) with embedded Julia
- Write Qt bindings for Julia and make all in Julia
- Use Qt with Python and the Python Julia bindings
- Use Gtk.jl
- Use Tk.jl

Each of these has another downside. You have to decide yourself, what compromises you want to make.
 

As far as Qt is concerned, wx is also viable alternative, but Qt seems
to be the best option today for multi-platform app due to having much
more developing activity than wx and, as I wrote earlier, wherever one
goes, nobody recommends using GTK for anything except Linux...actually I
just hear the word 'pain' mentioned often in regard.

Gour, we have understand that you are a Qt fan, and actually I like it too.
But please avoid saying insulting things about Gtk here. One of the core Julia
programmer (Jameson) has done awesome work to implement Gtk bindings and
I have done quite some work to fill missing pieces. julia-dev is not the place to
bash others technology.

[Gour]

On Mon, 17 Feb 2014 03:13:02 -0800 (PST)
Tobi <tobias...@googlemail.com> wrote:

> Gour, we have understand that you are a Qt fan, and actually I like
> it too. But please avoid saying insulting things about Gtk here. O

I'm not Qt fan, but just (trying to be) pragmatic here.

Most of my Linux days I spent with GTK technology and would be happy if
it would be in better shape multi-platform-wise.

> of the core Julia programmer (Jameson) has done awesome work to
> implement Gtk bindings and I have done quite some work to fill missing
> pieces.

It was certainly not meant to offend anyone personally and it's
glorious to work & provide *any* GUI bindings for Julia making it
attractive as 'general purpose' language'.

> julia-dev is not the place to bash others technology.

I apologize for it. It was never my intention.


Sincerely,
Gour

--
In the material world, one who is unaffected by whatever good
or evil he may obtain, neither praising it nor despising it,
is firmly fixed in perfect knowledge.

[Tobi]

Am Montag, 17. Februar 2014 12:02:50 UTC+1 schrieb Gour:

Well, afaik, it's still in the range of 2X.

My experience is that it is most of the time a factor of about 1.2. One often has to do some manual tweeking though (i.e. us @inbounds).
When the autovectorizer of gcc/llvm/intel kicks in the factor might be larger. But autovectorization will also land in Julia (there is already a
pull request for that). Static compilation will not change the execution speed.

 

I did not express clearly - I was not thinking about statically compiled
and linked application, but about the speed difference between using JIT
compiler and native code generation as it is e.g. between Julia and
Nimrod.

 
I don't want to start any comparison with Nimrod here (which looks like a really cool language by the way). But before thinking about whether
a statically compiled application is faster than a JIT compiled, one has to benchmark it. gcc can statically compile Java and it is not faster
than the regular JIT compiled Java.

The nice thing about Julia is that it uses the llvm JIT compiler which allows to dump generated functions into a shared library. This is done
in the 0.3 release and leads to a blazing fast startup. In the future we will see more of this for modules and maybe in the end even for Julia scripts.
This is a non-trivial process but I am totally impressed with the developement speed of Julia. Having statically compiled system image is a major achievment.

[Gour]

On Mon, 17 Feb 2014 05:23:56 -0800 (PST)
Tobi <tobias...@googlemail.com> wrote:

> My experience is that it is most of the time a factor of about 1.2.

That's really great and much more than I could expect.

> Static compilation will not change the execution speed.

Yeah, I got some explanation in #nimrod why it may be so for Julia.

[Francesco Bonazzi]

On Sunday, February 16, 2014 10:45:47 PM UTC+1, Tobi wrote:

What is the main motivation to have binary executables? Using the C-API it should be quite simple to write a small C program that links to libjulia and executes some code.

Well, there is a good reason indeed. Companies wishing to produce non-free closed-source software would prefer to distribute binary-compiled programs, in order to avoid their code to be stolen.

There are lots of companies developing in Java who choose to use compilers to machine code instead of the JVM for their final product, for the same reason (Java bytecode is easily decompilable).

I agree that having compilation to binary executable would be of small advantage for open source projects (maybe there's some minor advantage on resource-limited devices?).

[Stefan Karpinski]

This is certainly a use case. I'm sure you can see, however, why this isn't a very compelling use case for people donating their efforts to an open source project. There's not a lot of motivation to implement very tricky features for free so that other people can make more money.

[Tobi]

Am Montag, 17. Februar 2014 19:23:18 UTC+1 schrieb Stefan Karpinski:

This is certainly a use case. I'm sure you can see, however, why this isn't a very compelling use case for people donating their efforts to an open source project. There's not a lot of motivation to implement very tricky features for free so that other people can make more money.

Stefan, you are absolutely right that this does not seem to be of interest for the core Julia community on a first sight. My feeling as someone working in a company and still contributing to Julia core and Gtk.jl is that hiding the code is yet another big feature that Julia will have. And I am further quite confident that this feature will actually help Julia a lot. When I would distribute a closed source application I would want to hide a) my secret algorithm and b) the boring but development intensiv application logic. However, there is also c) which is the standard libraries (i.e. packages). And for these I would always contribute upstream.

When module precompilation lands this is IMHO a real selling point for Julia and this will make it very attractive for many companies that currently avoid languages like Python and Java.

[Stefan Karpinski]

That may well be. Personally, I think fully static compilation is primarily attractive because it drastically simplifies application deployment: put a single pre-compiled binary on a server and run it – end of story.

[Gour]

On Mon, 17 Feb 2014 16:52:33 -0500
Stefan Karpinski <ste...@karpinski.org> wrote:

> That may well be. Personally, I think fully static compilation is
> primarily attractive because it drastically simplifies application
> deployment: put a single pre-compiled binary on a server and run it –
> end of story.

I agree, especially for end-users on the OS-es where that is a
common practice for deploying.


Sincerely,
Gour

--
But for one who takes pleasure in the self, whose human life
is one of self-realization, and who is satisfied in the self only,
fully satiated — for him there is no duty.

[Fil Mackay]

On Tuesday, February 18, 2014 8:52:33 AM UTC+11, Stefan Karpinski wrote:

That may well be. Personally, I think fully static compilation is primarily attractive because it drastically simplifies application deployment: put a single pre-compiled binary on a server and run it – end of story.

Would the static executable be purely for deployment convenience, ie. would JIT at runtime identical to the current model by just embedding the AST (or text?) of the code in the executable file?

[Stefan Karpinski]

In that case, you'd probably want to pre-compile everything. There are other situations where you'd want to keep the JIT capability around.

[Fil Mackay]

On Tuesday, February 18, 2014 12:00:54 PM UTC+11, Stefan Karpinski wrote:

In that case, you'd probably want to pre-compile everything. There are other situations where you'd want to keep the JIT capability around.

I presume you could pre-JIT what you can, but still be able to JIT at runtime if required. The stuff I work on is all metaprogramming so relies on runtime eval. Deferring this to runtime JIT also ensures you can optimise to the actual CPU.

[Steven G. Johnson]

On Monday, February 17, 2014 8:00:54 PM UTC-5, Stefan Karpinski wrote:

In that case, you'd probably want to pre-compile everything. There are other situations where you'd want to keep the JIT capability around.

That's what I was thinking of as static compilation:   Precompile as much as possible, launch a small executable that loads this and goes, but still link to libjulia (including LLVM).   I don't think shipping libjulia is too much of a burden (unless you're on a tiny embedded system), as long as things are mostly precompiled so that you don't have a big startup latency.

Analyzing code to determine whether it requires runtime JIT is a much harder problem, and I don't see that being solved for a while.

[Tobi]

Am Dienstag, 18. Februar 2014 13:57:06 UTC+1 schrieb Steven G. Johnson:

That's what I was thinking of as static compilation:   Precompile as much as possible, launch a small executable that loads this and goes, but still link to libjulia (including LLVM).   I don't think shipping libjulia is too much of a burden (unless you're on a tiny embedded system), as long as things are mostly precompiled so that you don't have a big startup latency.

 
I second that. One has to ship some shared libraries (BLAS/LAPACK) anyway. Furthermore, when module precompilation lands one presumely haa to ship one more shared library for each package one is using.

[Jameson Nash]

How does this differ from what we do now with sys.so?

[Tobi]

Am Dienstag, 18. Februar 2014 19:16:29 UTC+1 schrieb Jameson:

How does this differ from what we do now with sys.so?

Not at all. I know that it is now possible to extend the sys.so with packages. But it seems to be cleaner when we have one .so per package/module.

This is what I understand under static compilation. The last step is to compile Julia scripts to binary applications. But IMHO this is not the most important thing. One can put everything in modules and only provide a trivial loader script. 

[Steven G. Johnson]

On Tuesday, February 18, 2014 1:16:29 PM UTC-5, Jameson wrote:

How does this differ from what we do now with sys.so? 

It might be nice to avoid the julia executable entirely, and provide an easy way to generate a little loader executable.  But you're right that a trivial little shell script which calls /usr/bin/julia -J with a custom sysimage file should be sufficient.