[shen-scheme] Chez Scheme support / Does anyone care about shen-scheme supporting chibi/Gauche?

[Bruno Deferrari]

I rewrote the Kl->Scheme compiler of shen-scheme in Shen and made shen-scheme work with Chez.

I haven't started working on performance optimizations yet and the test suite completes in 3.3 seconds on my machine (for reference, the SBCL port completes the tests in 4.6 seconds and the Gauche Scheme port did in 9.8 seconds). This is with no peephole optimizations and by running everything in Chez' interactive environment that doesn't optimize as much as it could because it assumes every binding is mutable. That means that from here, it is only going to get even faster. Chez is quite impressive.

But right now neither chibi-scheme or Gauche are able to run this new version of shen-scheme.

The question is, does anybody care about that?

If not, I'm considering making Chez the only supported Scheme implementation for a while to focus on improving it. I may still add back support for chibi/Gauche (and other Scheme implementations) in the future once I'm happy with the Chez version.

--

BD

[Mark Tarver]

That's really impressive; well done.  

Mark

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.
To post to this group, send email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

[Racket Noob]

That's wonderful news, Bruno! Congratulations!

I think that it's much more important to optimize chez version of Shen to be as fast as possible. Chibi and Gauche version seems much less important to me.

Chez is the best scheme implementation around. Even those pathetic academic-paper-massproducing people from the PLT group have recently considered chez scheme as a new core engine for their slow Racket academic sandbox.

RN

[deech]

Congrats! I'm happy with having just Chez.

[Bruno Deferrari]

Ok, great. I will probably try to get chibi working again in the future because I find it very useful for finding places where the Shen kernel may make things hard (performance wise) to underlying platforms that are either interpreted or that don't have very advanced compilers.

Btw, after adding back the same peephole optimizations I had before the rewrite, now the time to run the test suite completes in 1.8 seconds (down from 3.3). This is while still not fully taking advantage of the performance optimizations Chez can provide (still running everything in Chez interactive environment).

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.
To post to this group, send email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

[Mark Tarver]

That is some amazing speed.   What is your processor?

Mark

[Bruno Deferrari]

A 2.4 GHz Intel Core i5 (of a macbook pro from late 2013, if I'm not wrong "Haswell" was Intel's newest microarchitecture at that time, so mine is probably that one).

[Mark Tarver]

Ah;  sent that off before reading closely sorry;  you gave the times above.  You changed the code as well so some of it is you and some Chez I guess.   Good work.

Mark 

On Monday, May 22, 2017 at 9:42:47 AM UTC+1, Mark Tarver wrote:

Very impressive; what were the times for Gauche and Chibi?   Is this all down to Chez?

Mark

[Bruno Deferrari]

On Mon, May 22, 2017 at 5:49 AM, Mark Tarver <dr.mt...@gmail.com> wrote:

Ah;  sent that off before reading closely sorry;  you gave the times above.  You changed the code as well so some of it is you and some Chez I guess.   Good work.

It is 90% Chez. The Kl->Scheme compiler generates good Scheme code and avoids as much unnecessary indirection as it can, but that can only get you performance that is close to what you would get when writing plain Scheme -- the underlying platform still has to be fast, and Chez is very fast, more than I was expecting (and I had high expectations).

Gauche ~10 seconds and chibi ~33 seconds. The code the new compiler generates is almost the same as the code the old one generated, it is just that Chez is much faster than Gauche and chibi.

There are some things I tried before but that didn't work too well that I have to try now because Chez can probably handle it well, like pattern factorization, which I tried using internal function defines before to see if chibi was smart enough to convert those to direct jumps, but it wasn't the case -- but Chez may be able to.

If I make it so that functions defined in the "shen" package are not overrideable I can probably bring the numbers down further (this would let me move all the kernel code to an immutable environment, where the Chez compiler can perform a bunch of optimizations that it is not doing right now).

--

BD

[Mark Tarver]

Sounds like Chez is the real deal.  It seems Dybvig from Indiana University is the guy who built this.  I'll write to him and give him congratulations.  But good for you for getting Shen onto Chez.

Pattern factorisation can offer significant speedups - see http://www.shenlanguage.org/Professional/Help/griffon/.

I haven't run the SP kernel through Griffon; speed is not my focus right now.

However the old Qi used factorisation and beat the Lisp hackers for speed.  

Another point of optimisation, not explored, is between the algebraic sequent calculus and the generated code.  ASC allows you to predict the needed structure of any type secure input,   and aovid unneeded tests.

With effective use of the type information that Shen produces, and modern compiler technology, one could make Shen blazingly fast.

Mark

[Pierpaolo Bernardi]

On Mon, May 22, 2017 at 10:42 AM, Mark Tarver <dr.mt...@gmail.com> wrote:
> Very impressive; what were the times for Gauche and Chibi? Is this all
> down to Chez?

here's a suite of general benchmark results for various schemes:
https://ecraven.github.io/r7rs-benchmarks/benchmark.html

Gauche and Chibi, as nice as they are, are not speed demons.

[deech]

Chez is incredibly interesting. Not only does it produce fast code but the compiler itself is fast. Dybvig reportedly avoided the trap of the compiler getting slower and slower over time by ruthlessly rejecting features that increased it. I think he also pioneered the technique of transforming source to machine code using many (100's?) passes instead of just 5-10 like most compilers. I was very happy when it was OS'ed and a huge thanks to Bruno for writing this port.

Bruno, what is the Windows install story like?

-deech

 

Mark 

Mark

To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To post to this group, send email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

--
You received this message because you are subscribed to the Google Groups "Shen" group.

To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To post to this group, send email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Shen" group.

To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To post to this group, send email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

--
You received this message because you are subscribed to the Google Groups "Shen" group.

To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To post to this group, send email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

--
You received this message because you are subscribed to the Google Groups "Shen" group.

To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

[Bruno Deferrari]

On Mon, May 22, 2017 at 9:59 AM, Mark Tarver <dr.mt...@gmail.com> wrote:

Sounds like Chez is the real deal.  It seems Dybvig from Indiana University is the guy who built this.  I'll write to him and give him congratulations.  But good for you for getting Shen onto Chez.

Pattern factorisation can offer significant speedups - see http://www.shenlanguage.org/Professional/Help/griffon/.

Yes, the hard part here is that Scheme has nothing like Common Lisp labelled jumps. What I tried instead back then was using internal defines (defining functions inside a top level function) in place of labels, hoping the underlying compiler was smart enough to optimize that (the OCaml compiler does, chibi doesn't, I'm not familiar enough with Chez yet to tell if it does).

 

I haven't run the SP kernel through Griffon; speed is not my focus right now.

However the old Qi used factorisation and beat the Lisp hackers for speed.  

Another point of optimisation, not explored, is between the algebraic sequent calculus and the generated code.  ASC allows you to predict the needed structure of any type secure input,   and aovid unneeded tests.

With effective use of the type information that Shen produces, and modern compiler technology, one could make Shen blazingly fast.

That is quite a bit more work. I played a bit with that when I was working on a (never finished) OCaml port. There was no way to take advantage of that in chibi or Gauche, but there is in Chez (which allows you to write function calls in a way that bypasses runtime checks, but you have to make sure your code is type safe).

Example:

> (define (test a b)
    (#3%fx+ a b))
> (test 1 2)
3
> (test 1 2.3)
0.0
> (test 2 3.4)
2.1655232955e-314|33

The `fx+` function adds integers, and only integers. The `#3%` prefix makes it so that the call is compiled as if optimization level 3 (the highest level in Chez) was enabled. Optimization level 3 makes code faster (and unsafe) by disabling runtime checks (as you can see in the previous example).

Anyway, it will probably be a while before I go down that road, there are easier (and safer) things to try first.

--

BD

[Bruno Deferrari]

On Mon, May 22, 2017 at 10:32 AM, deech <aditya...@gmail.com> wrote:

Chez is incredibly interesting. Not only does it produce fast code but the compiler itself is fast. Dybvig reportedly avoided the trap of the compiler getting slower and slower over time by ruthlessly rejecting features that increased it. I think he also pioneered the technique of transforming source to machine code using many (100's?) passes instead of just 5-10 like most compilers. I was very happy when it was OS'ed and a huge thanks to Bruno for writing this port.

Bruno, what is the Windows install story like?

I don't know, but from what I have seen Chez fully supports Windows, so I don't think there will be a problem there. Chez can also produce standalone programs.

The only place where I think there may be issues is when adding features that interface with the operating system (like support for sockets for example), but as long as there are Windows users willing to test things, it shouldn't be an issue.

 

To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.

To post to this group, send email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

[Raoul Duke]

If you can target Lambda Native then Shen is a mobile app dev language.

[Bruno Deferrari]

I was not able to do a release during the weekend like I had planned, but for anyone wanting to play with it, the Chez version can be found in the "chez" branch of the shen-scheme repository:

https://github.com/tizoc/shen-scheme/tree/chez

It is slightly faster than last time, but not as much since I haven't figured a good way to enable the extra optimizations provided by Chez in a way that doesn't break Shen, and neither was I able to use full-program-optimization in a way that is compatible with Shen's `eval`. I may reserve that for when batch compilation mode for programs that don't make use of eval is added.

Another thing I haven't figured out yet is how to generate a redistributable program. Creating a Shen boot image for Chez almost works, except that when doing so command line arguments don't work (the list is always empty) and it seems there is no way to get rid of Chez Scheme's own command line flags. The README instructions instead describe how to generate a "compiled script" that can then be launched using Chez, or in the case of Unix systems, just used as an executable by having a shebang line at the beginning of the file that points to the Chez executable.

Btw, anyone trying to write a Shen compiler in Shen itself (instead of the host language) may be interested in taking a look at the "build.shen" script to see how it is done in shen-scheme:

https://github.com/tizoc/shen-scheme/blob/chez/scripts/build.shen

On Mon, May 22, 2017 at 11:06 AM, Raoul Duke <rao...@gmail.com> wrote:

If you can target Lambda Native then Shen is a mobile app dev language.

--

You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.
To post to this group, send email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD