Wasp Lisp port of Shen

[Chris Double]

I've have a work in progress port of Shen to Wasp Lisp, heavily based
on the Scheme ports. It's available on github at:

https://github.com/doublec/shen-wasp

Wasp is a small lisp with concurrency and the ability to send bytecode
to other lisp processes. This is used in the MOSREF framework:

https://bluishcoder.co.nz/2009/11/28/using-wasp-lisp-secure-remote-injection.html

I use Wasp for various things and wanted to play with the same ideas
but with Shen, hence the port.

It's still a work in progress. All but one of the tests pass. The
Proof Assistant file is the one that fails. It either hangs or takes
longer than I've been willing to wait to load.

It hangs on the 'globals' definition below:

-------------------------8<-----------------
(synonyms

proof (list step)
step ((list sequent) * tactic)
tactic ((list sequent) --> (list sequent))
sequent ((list wff) * wff))

(datatype globals

_______________________
(value *proof*) : proof;)
-------------------------8<-----------------

It seems to be stuck in the recursive calls to shen.mu_reduction. I'll
do more debugging to track it down.

It's pretty slow - I need to do some optimizing of both the
compilation to Wasp code and optimization on the Wasp end. I have a
static binary that should run on any Linux 64 bit machine here:

https://bluishcoder.co.nz/shen/shen_static.bz2

Run with:

$ bunzip2 shen_static.bz2
$ chmod +x shen_static
$ ./shen_static

--
http://bluishcoder.co.nz

[Bruno Deferrari]

Thats great. I see that because of the lack of the "match" library you had to translate the pattern matching code to a cond with predicates, I bet that was.... "fun" :)

--
http://bluishcoder.co.nz

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.
To post to this group, send an email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

[Mark Tarver]

Well done;  the list of official ports now includes for the first time a language I didn't even know existed!

Seriously though, congratulations.

bw

Mark 

--
http://bluishcoder.co.nz

[Chris Double]

On Mon, Apr 24, 2017 at 3:54 AM, Bruno Deferrari <uti...@gmail.com> wrote:
> Thats great. I see that because of the lack of the "match" library you had
> to translate the pattern matching code to a cond with predicates, I bet that
> was.... "fun" :)

Yes, the joys of Lisp's without the neat tools in modern Schemes and
such, much tearing of hair out! "You are lost in a twisty maze of car,
cdr, cadr, and caddr's".

In hindsight it might have been easier to write the compiler part in
Shen and run it on one of the other Shen implementations to bootstrap
it. Then I could use all the Shen pattern matching goodies.

--
http://bluishcoder.co.nz

[Bruno Deferrari]

Indeed, writing it in Shen is the best choice. I had done so for the Scheme port a while back, but I never merged it back or pushed into the repo, and I lost that code. I have plans to do it again in the future, because it will make it much easier for me to support other Scheme implementations.

 

--
http://bluishcoder.co.nz

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.
To post to this group, send an email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

[Chris Double]

On Sun, Apr 23, 2017 at 10:59 PM, Chris Double
<chris....@double.co.nz> wrote:
>
> I have a static binary that should run on any Linux 64 bit machine here:
>
> https://bluishcoder.co.nz/shen/shen_static.bz2

A Windows build is available now at:

https://bluishcoder.co.nz/shen/shen.zip

The zip file contains the 'shen.exe' which should run on 64 bit
Windows systems. Both the Linux and Windows build are updated to the
20.0 kernel.

--
http://bluishcoder.co.nz

[Bruno Deferrari]

Chris, did you happen to compare the performance of the Wasp port, before, and after upgrading to Shen OS Kernel 20.0 ?

--
http://bluishcoder.co.nz

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.
To post to this group, send an email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

[Chris Double]

On Tue, Apr 25, 2017 at 1:43 AM, Bruno Deferrari <uti...@gmail.com> wrote:
> Chris, did you happen to compare the performance of the Wasp port, before,
> and after upgrading to Shen OS Kernel 20.0 ?

I'll do a comparison and see what the result is. I still can't run the
"proof assistant" test, but can test with the others. I haven't done
any overriding of the new functions suggested for 20.0 dict's support
yet.

--
http://bluishcoder.co.nz

[Chris Double]

19.3.1 ran in 810 seconds, 20.0 ran in 710 seconds, so a nice speed up there.

--
http://bluishcoder.co.nz

[Bruno Deferrari]

Thank you.

I'm trying to build your fork of WaspVM on OSX but I'm getting errors (because of functions defined inside of other functions). I guess it doesn't build with clang and I need gcc?

Btw, you will not be able to do `(track shen.mu_reduction)` because the functions that come in shen don't have their klambda code associated at runtime, but you can easily solve that by adding this to a file, then loading it with `load`:

(package shen []

\\ copypaste the source code of mu_reduction in prolog.shen here

)

after you load this file you can do `(load "mu_reduction.shen")` and `(track shen.mu_reduction)`.

This is going to show all the inputs and outputs that go through that function.

I just tested it, and the max recursion depth is 5 for that "globals" datatype definition. You can see the output here: https://gist.githubusercontent.com/tizoc/d2bfa599bb03768ff77c9eaca3d07206/raw/73db2927b8db7dca9e11b46e8fa434616fb99115/gistfile1.txt

Hope this helps.

 

--
http://bluishcoder.co.nz

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.
To post to this group, send an email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

[Chris Double]

On Tue, Apr 25, 2017 at 12:36 PM, Bruno Deferrari <uti...@gmail.com> wrote:
>
> I'm trying to build your fork of WaspVM on OSX but I'm getting errors
> (because of functions defined inside of other functions). I guess it doesn't
> build with clang and I need gcc?

Yes, the VM needs GCC because it uses nested functions - a GCC
extension. I used to develop on Wasp on Mac OS X so it used to build
cleanly at some point.

>
> I just tested it, and the max recursion depth is 5 for that "globals"
> datatype definition. You can see the output here:
> https://gist.githubusercontent.com/tizoc/d2bfa599bb03768ff77c9eaca3d07206/raw/73db2927b8db7dca9e11b46e8fa434616fb99115/gistfile1.txt

That's great, thanks for the tips! I'm hoping it's something simple.
The previous issues I came across were related to errors being thrown
by Wasp and caught in a "trap-error" in the klambda code and silently
discarded so it might be something similar.

--
http://bluishcoder.co.nz

[Chris Double]

On Tue, Apr 25, 2017 at 12:36 PM, Bruno Deferrari <uti...@gmail.com> wrote:
>

> I'm trying to build your fork of WaspVM on OSX but I'm getting errors
> (because of functions defined inside of other functions). I guess it doesn't
> build with clang and I need gcc?

I just tested with gcc installed via homebrew. This built Wasp:

$ CC=gcc-5 make install

I put a Mac OS Shen binary at https://bluishcoder.co.nz/shen/shen_macos.bz2

It was slower to start than on my Linux machine but my Macbook Air is
one of the first generations so slower than my Linux box.

--
https://bluishcoder.co.nz

[Chris Double]

I fixed a bug in the FFI for accessing Wasp internal functions and it
now works like the Scheme ports. 'wasp." takes a string that is
evaluated as Wasp code, or "wasp.foo" is a symbol in Wasp called
"foo". An example of spawning a thread:

(wasp.spawn (freeze (do (wasp.pause 5000) (print "Hello World\n"))))

This spawns a thread that waits for 5 seconds then prints the string.
The Wasp function 'spawn' takes a lambda that has no arguments. I
couldn't find a way to create such a thing from Shen, but "freeze" is
implemented to wrap its expression in a no argument lambda so that
worked for this case.

Note that Wasp threads are cooperative, not preemptive, and are green
threads, not system threads.

--
http://bluishcoder.co.nz

[Mark Tarver]

Well done on Shen OS 20!.

Mark

[Chris Double]

On Tue, Apr 25, 2017 at 12:36 PM, Bruno Deferrari <uti...@gmail.com> wrote:

> I just tested it, and the max recursion depth is 5 for that "globals"
> datatype definition. You can see the output here:
> https://gist.githubusercontent.com/tizoc/d2bfa599bb03768ff77c9eaca3d07206/raw/73db2927b8db7dca9e11b46e8fa434616fb99115/gistfile1.txt

I just tried this and I get something completely different. I've
duplicated your results under shen-c so I'm doing something wrong
somewhere. There very first mu_reduction looks like:

[[shen.mu [mode [[value *proof*] : [mode [list [[list [[list wff] *
wff]] * [[list [[list wff] * wff]] --> [list [[list wff] * wff]]]]]
+]] -] [[shen.mu [mode Context_1957 +] [shen.pop shen.the shen.stack]]
V1359]] V1358], +, ==>

Compared to your:

[[shen.mu [mode V1189 +] [[shen.mu [mode V1190 +] [shen.rename
shen.the shen.variables in [Context_1957] and shen.then [call shen.the
shen.continuation [[unify [cons V1189 [cons V1190 []]] [cons [cons
[cons value [cons *proof* []]] [cons : [cons [cons list [cons [cons
[cons list [cons [cons [cons list [cons wff []]] [cons * [cons wff
[]]]] []]] [cons * [cons [cons [cons list [cons [cons [cons list [cons
wff []]] [cons * [cons wff []]]] []]] [cons --> [cons [cons list [cons
[cons [cons list [cons wff []]] [cons * [cons wff []]]] []]] []]]]
[]]]] []]] []]]] [cons Context_1957 []]]]]]]] V1192]] V1191], +, ==>

It then recurses 74 deep and where I break out.

--
http://bluishcoder.co.nz

[Chris Double]

On Tue, Apr 25, 2017 at 10:14 PM, Chris Double
<chris....@double.co.nz> wrote:
>
> It then recurses 74 deep and where I break out.

I found the problem. The shen.head_abstraction function has a clause
that checks if the shen.complexity_head < (value *complexitymax*). On
my port it's taking this branch. In other ports it's taking the other
branch. The reason the Wasp port is handling it differently is that
there is an integer overflow in computing the complexity for the size
of wasp integers. The complexity comes out at 576460752303423488. Wasp
was originally a 32 bit Lisp and retains the integer size there. I'll
be patching Wasp for a fix.

Is there a danger of that complexity overflowing on other systems that
are 64 bit but don't have arbitary precision integers?

--
http://bluishcoder.co.nz

[Chris Double]

The Shen documentation
(http://www.shenlanguage.org/learn-shen/shendoc.htm) states:

"The maximum size of any integer or float and the precision of the
arithmetic is implementation dependent."

But the complexity result being returned is greater than a 32-bit Shen
can handle. This limits the ability to run on embedded devices. Can
that function be modified to result in a complexity that fits within
32-bits? Or if the tests define a conforming implementation should the
limit be mentioned in the documentation?

--
http://bluishcoder.co.nz

[Bruno Deferrari]

I think the function can be changed. The final value of `complexity_head` is not important, what is important is to know if it is greater than the value of `*maxcomplexity*`.

So intead of  `(< (complexity_head H) (value *maxcomplexity*))` the test can be something like `(product-greater-than H (value *maxcomplexity*))`, with `product-greater-than` being like `product`, but it returns a boolean and shortcircuits once it knows that the product of the values in the list `H` is greater than its last argument.

If you wait I can give a try later today, or you can try it yourself if you are in a hurry (you can use the same `load` trick I mentioned earlier to override `head_abstraction` at runtime, make sure to wrap it in `(package shen [] ...)`).

 

--
http://bluishcoder.co.nz

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.
To post to this group, send an email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

[Bruno Deferrari]

On Tue, Apr 25, 2017 at 8:57 AM, Bruno Deferrari <uti...@gmail.com> wrote:

On Tue, Apr 25, 2017 at 8:32 AM, Chris Double <chris....@double.co.nz> wrote:

On Tue, Apr 25, 2017 at 11:22 PM, Chris Double
<chris....@double.co.nz> wrote:
>
> Is there a danger of that complexity overflowing on other systems that
> are 64 bit but don't have arbitary precision integers?
>

The Shen documentation
(http://www.shenlanguage.org/learn-shen/shendoc.htm) states:

"The maximum size of any integer or float and the precision of the
arithmetic is implementation dependent."

But the complexity result being returned is greater than a 32-bit Shen
can handle. This limits the ability to run on embedded devices. Can
that function be modified to result in a complexity that fits within
32-bits? Or if the tests define a conforming implementation should the
limit be mentioned in the documentation?

I think the function can be changed. The final value of `complexity_head` is not important, what is important is to know if it is greater than the value of `*maxcomplexity*`.

So intead of  `(< (complexity_head H) (value *maxcomplexity*))` the test can be something like `(product-greater-than H (value *maxcomplexity*))`, with `product-greater-than` being like `product`, but it returns a boolean and shortcircuits once it knows that the product of the values in the list `H` is greater than its last argument.

Sorry, haven't had my morning coffee yet. It would be `product-less-than`.

 

If you wait I can give a try later today, or you can try it yourself if you are in a hurry (you can use the same `load` trick I mentioned earlier to override `head_abstraction` at runtime, make sure to wrap it in `(package shen [] ...)`).

 

--
http://bluishcoder.co.nz

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.
To post to this group, send an email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

--

BD

[Bruno Deferrari]

On Tue, Apr 25, 2017 at 8:58 AM, Bruno Deferrari <uti...@gmail.com> wrote:

On Tue, Apr 25, 2017 at 8:57 AM, Bruno Deferrari <uti...@gmail.com> wrote:

On Tue, Apr 25, 2017 at 8:32 AM, Chris Double <chris....@double.co.nz> wrote:

On Tue, Apr 25, 2017 at 11:22 PM, Chris Double
<chris....@double.co.nz> wrote:
>
> Is there a danger of that complexity overflowing on other systems that
> are 64 bit but don't have arbitary precision integers?
>

The Shen documentation
(http://www.shenlanguage.org/learn-shen/shendoc.htm) states:

"The maximum size of any integer or float and the precision of the
arithmetic is implementation dependent."

But the complexity result being returned is greater than a 32-bit Shen
can handle. This limits the ability to run on embedded devices. Can
that function be modified to result in a complexity that fits within
32-bits? Or if the tests define a conforming implementation should the
limit be mentioned in the documentation?

I think the function can be changed. The final value of `complexity_head` is not important, what is important is to know if it is greater than the value of `*maxcomplexity*`.

So intead of  `(< (complexity_head H) (value *maxcomplexity*))` the test can be something like `(product-greater-than H (value *maxcomplexity*))`, with `product-greater-than` being like `product`, but it returns a boolean and shortcircuits once it knows that the product of the values in the list `H` is greater than its last argument.

Sorry, haven't had my morning coffee yet. It would be `product-less-than`.

Well, it turns out it is a bit more complicated than that, because the result of `complexity` can be huge too:

 <1> Inputs to shen.product-less-than-h
   [2 576460752303423488 1], 128, 1,  ==>

 I created a new issue for this on Github: https://github.com/Shen-Language/shen-sources/issues/34

 

If you wait I can give a try later today, or you can try it yourself if you are in a hurry (you can use the same `load` trick I mentioned earlier to override `head_abstraction` at runtime, make sure to wrap it in `(package shen [] ...)`).

 

--
http://bluishcoder.co.nz

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.
To post to this group, send an email to qil...@googlegroups.com.
Visit this group at https://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

--

BD

--

BD

--

BD

[Chris Double]

On Wed, Apr 26, 2017 at 12:39 AM, Bruno Deferrari <uti...@gmail.com> wrote:
>
> Well, it turns out it is a bit more complicated than that, because the
> result of `complexity` can be huge too:
>
> <1> Inputs to shen.product-less-than-h
> [2 576460752303423488 1], 128, 1, ==>
>
> I created a new issue for this on Github:
> https://github.com/Shen-Language/shen-sources/issues/34

Thanks, I did a pull request as a possible fix:
https://github.com/Shen-Language/shen-sources/pull/35

The proof assistant test loads and runs with this. It changes
complexity to raise an error on overflow and fail the *maxcomplexity*
check in that case.

--
http://bluishcoder.co.nz

[Mark Tarver]

I've put this up under your name.  Thx for the contribution.

Mark