disappeared links?

[dr.mt...@gmail.com]

I wanted to see the old ports of Shen on https://shen-language.github.io/ 

and found they had all gone.  The history of this site

is that Robert Koeniger didn't like the look of the old

official site -and wanted a place where he and other

open sourcerers could do their own thing.  So he 

copied the info from the download page on the

official site and set up shop on https://shen-language.github.io/

in 2018 as I recall.

I wasn't certain this was a great move, but I was on holiday

when he moved on this and from that time I dropped keeping a

 list and pointed to the github site.   So the list of implementations

grew on the github site until they seem to have gone.

I have no control over this site and I rarely look at it.  I only did

so because Jacob is working on an F# and I wondered why, since

Robert did the heavy lifting in porting Shen to F#, why not just 

patch Robert's version?  Hence I looked and its gone.

I think there are sound reasons for drawing a line under a lot of

this old github work.  It is nearly all pre-S kernel implementations.

The certificates of warranty  placed against these old versions are 

misleading because they suggest conformity to the post 2021 

standards but .....

First they do work and the greater majority of Shen programs still 

run on them.   Second the amount of work needed to bring them to

scratch is not great.  Actually I think it might be a matter of throwing

away code because the S kernel takes over some of the weight from

the porter e.g. partial applications.

I think a health warning needed to be placed over these links but

they were important and valuable records of work.  

I do not have the list of ports after 2018 when I handed over control.   Also

the WayBack machine seems not to work on the github site so 

there is no public record of what was there.  The Shen site does

respond to WayBack.

It might be a good idea to have the links back under a health warning in

github if they can be retrieved.    I appreciate the pointer to the official

site in the new site but I don't have the information on my site to point

people towards.  If wanted I can take this data back under my wing.

Mark

[Bruno Deferrari]

Hi Mark, the raw data used to construct that list is here: https://github.com/Shen-Language/shen-language.github.io/blob/master/data.shen

It was then rendered using this code which executed inside the user's browser: https://github.com/Shen-Language/shen-language.github.io/blob/master/index.shen and https://github.com/Shen-Language/shen-language.github.io/blob/master/fetch.shen

I can bring it back to the page but in a very simplified form (active vs historical ports), and hardcoded so that it is not rendered dynamically (it was a pretty cool example of using ShenScript in the browser, but requiring Javascript + queries to GitHiub API just to render a table made it less compatible and more fragile).

Btw, what Jacob is updating Robert's code, he is not starting from scratch.

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/1046751f-06c5-4f7a-af82-86ca7e10726an%40googlegroups.com.

--

BD

[Bruno Deferrari]

Table is back up. No javascript used to render it now, no versions mentioned, and the split is between active and inactive (right now only the official Common Lisp port and Shen/Scheme are in the active section, I expect ShenSharp to move over there soon too once Jacob is happy with it).

--

BD

[dr.mt...@gmail.com]

Yes, the new arrangement seems very logical

since the pre-S versions are offset from

the current ones.  Does the Go version

count as current because as AFAIK it is

based on an S kernel?

I didn't know that Shen was used to render

the github site.  That's very clever and deserves

a wider application.  I didn't see anything needing

that tech in the site.  It would seem overkill, 

I agree, for one table.  And there is only one 

page?   I used Shen only because the

official site became massive and I was forced to.

There's been a lot of traffic on the new F# version

which is great.  I assumed it was a completely

new build because I thought that reissuing 

Robert's work under the S kernel should be fairly

easy.   For the CL port of the S series I really just 

threw away code from the backend I didn't need to 

use any more.  

But it seems from what Jacob is writing

that it is not so easy as I thought for whatever reason.

The files he mentions are not ones I wrote.

I wrote the S series in part to be easier to port and

am interested in any challenges people face

in getting this to work for them.

Mark

[Bruno Deferrari]

On Wed, May 15, 2024 at 4:15 PM dr.mt...@gmail.com <dr.mt...@gmail.com> wrote:

Yes, the new arrangement seems very logical

since the pre-S versions are offset from

the current ones.  Does the Go version

count as current because as AFAIK it is

based on an S kernel?

I am not sure, but the last update on that repository was 2 years ago:

https://github.com/tiancaiamao/shen-go

 

I didn't know that Shen was used to render

the github site.  That's very clever and deserves

a wider application.  I didn't see anything needing

that tech in the site.  It would seem overkill, 

I agree, for one table.  And there is only one 

page?   I used Shen only because the

official site became massive and I was forced to.

The HTML iself is static, but once the page was loaded by the browser, some Shen code was executed (inside the browser, by being executed by ShenScript) to render some tables (list of ports, and list of contributors which is gone now).

It esd not required, but it was a cool ShenScript usage example that Robert implemented.

 

There's been a lot of traffic on the new F# version

which is great.  I assumed it was a completely

new build because I thought that reissuing 

Robert's work under the S kernel should be fairly

easy.   For the CL port of the S series I really just 

threw away code from the backend I didn't need to 

use any more.  

But it seems from what Jacob is writing

that it is not so easy as I thought for whatever reason.

The files he mentions are not ones I wrote.

I wrote the S series in part to be easier to port and

am interested in any challenges people face

in getting this to work for them.

Keep in mind that Common Lisp is very different from other environments. Being able to just load the code and dump an image simplifies the work quite a bit. But on platforms where you cannot do that, there are extra steps that are required to be able to precompile all the code ahead of time and to also restore the runtime state required by Shen. The init.kl file is mostly about solving this, without it the porter has to repeat the same process on his own or find some other workaround.

Speaking of which, I would prefer for this to be handled in the original sources release and not require the conversion done by make.shen in the GitHub release. The requirement is for every non-defun expression that shows up in the sources to be inside a function ("shen.init" maybe) that optionally (Common Lisp would not require this) gets called at startup.

 

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/442ac5d4-845b-47ec-b441-47db69050c80n%40googlegroups.com.

--

BD

[dr.mt...@gmail.com]

Keep in mind that Common Lisp is very different from other environments. Being able to just load the code and dump an image simplifies the work quite a bit. 

Quite true.  But there is one thing about Shen is that the kernel is very small.

For example,  under SBCL installing the kernel and the standard library takes

3 seconds.  In principle I could just skip saving an image and have a startup

that installs everything.  Maybe in other languages this is not true.  Anyway

I'm not going to second guess here because I've never used F#.

One thing I'm impressed with is that  Shen has been implemented in languages

like F# and Haskell.  I thought when I wrote Shen that it would only be supported

by languages that were dynamically typed since getting it into a Hindley-Milner

language would be akin to fitting a quart into a pint pot.  But people have done

what I thought would have been impossible.  Kudos.

M.

[dr.mt...@gmail.com]

In fact, I'd add that if you look at the OS work cited

on the github site, that it would without doubt

have merited academic publication and would

still merit it if people wanted to do it.   There

is a wealth of ingenuity in that work.  However

none of us, including myself, are really in that 

games.  It's a shame really because I've seen work

of far lesser quality funded for hundreds of 

thousands of pounds back in the 90s.   

M.

[Jacob MacDonald]

Dr. Mark Tarver wrote:
> But it seems from what Jacob is writing
> that it is not so easy as I thought for whatever reason.

The easy parts were quite easy, thanks to the hard work put in by you,
Robert, and the maintainers of open source .NET. Most of it comes down
to me being slow and unfamiliar with F# and Shen. But I figure I'll
lay out the timeline of my changes so I don't forget.

Six-ish years ago I got an inkling to play with Shen on .NET. This was
around the time that .NET Core was new, and ShenSharp supported
Framework and therefore only Windows (at least I remember it
misbehaving on Mono). I wanted it on Linux, and I wanted it on .NET
Core. I tried to figure out the upgrade, but misunderstood both the
tooling and ShenSharp's architecture. Robert helped me out but my
attempt was ultimately unsuccessful.

In the interim, Robert refactored the codebase to mostly work with
.NET Core. Meanwhile, the distinction between .NET Core and Framework
went away, and it became simpler to target all of .NET at once. At the
same time, F# became more tightly integrated with the new .NET
tooling; It had always been developed more openly than C#, so the
opening of C# let the projects gradually merge.

Last November, I took another look at the project and decided to take
a crack with the new tooling. The work I ended up doing mostly had
nothing to do with Shen, just with getting the project to build in a
Linux container. The commits from that period resolve a few bugs in
how we emit code, because ShenSharp originally built a DLL out of
translated KL in-memory for the runtime to use. This feature caused
all kinds of versioning grief and was ultimately removed from F#'s
compiler library, so I created a step similar to one in ShenScript
which saves the code to a file which can be included in a normal build
of the rest of the project. But the bulk of them have to do with
upgrading our use of F#'s compiler API, FSharp.Compiler.Service. That
library releases new versions as new versions of the language are
released, and there are sometimes breaking changes (there was a major
namespace overhaul and pieces of syntax accumulate new fields in them
for metadata; If ML-style types are familiar to you, it's no surprise
that this forces you to change two source locations for each such
addition). .NET library authors have the bad habit of relying on your
IDE's auto-complete instead of writing comprehensive changelogs, which
I find annoying and slowing. I finally got the REPL booting, but
something was broken and I got garbled output, so I shelved the
project for a while.

Last week I decided to fix that small issue and see how hard porting
to S series would be. The issue of newlines turned out to be another
tiny bug in the F# emitter, and the number of errors importing the S
series files was tractable. I had to update some F# calls of Shen
functions to match renames: add-macro became record-macro with a
slightly different signature and the test suite no longer needed to
load README.shen and then another file. I added the stub functions for
I/O: ShenSharp remains purely byte-oriented.

Those changes were (and the other ones are) small, but now I had to
figure out why Shen was breaking. I was slow because I find ML-style
languages hard to debug sometimes, and these bugs required unwinding
both Shen code and the execution tree of the ShenSharp interpreter. I
also misunderstood the nature of the GitHub sources as detailed in my
earlier messages to the group. ShenSharp's custom builtins were not
recognized by Shen's (fn) since ShenSharp does not update the property
vector: I was able to solve that using a leftover step that sets up
function arities and types after the kernel loads. There was a simple
bug in the interpreter's evaluator which meant that let did not work
at all inside lambdas or freezes: This is easy to miss because
functions get walked differently and I'm not sure if the issue existed
when evaluating pre-S-series Shen, it's not directly exercised by the
test suite; The fix was easy once I found it buried in Prolog
evaluation. And finally I realized that (type) was having its right
side walked, leading to function-not-found errors: That was buried in
YACC code.

[Jacob MacDonald]

Jacob MacDonald wrote:
> The easy parts were quite easy, thanks to the hard work put in by you,
> Robert, and the maintainers of open source .NET.

I have to correct myself and include Bruno here. I did not understand
the nature of the GitHub sources, which with the pointers from this
week are welcome.

[Jacob MacDonald]

Dr. Mark Tarver wrote:
> One thing I'm impressed with is that Shen has been implemented in languages
> like F# and Haskell. I thought when I wrote Shen that it would only be supported
> by languages that were dynamically typed since getting it into a Hindley-Milner
> language would be akin to fitting a quart into a pint pot. But people have done
> what I thought would have been impossible. Kudos.

The trick there is that it's barely a compiler: ML-family languages
are actually fairly straightforward to build symbolic interpreters in,
and Shen does all the type-checking so you only have to worry about
KLambda. ShenSharp only has 10 value cases and 13 expression cases,
and based on those it just walks the AST of whatever KLambda you give
it and evaluates against a global and local environment state. Simple
enough that even I can wrap my head around it, and my knowledge of
compilers is basically baby's-first-arithmetic-evaluator. All interop
is via boxing and unboxing CLR values. There's a major optimization
opportunity there: Even the generated F# for the kernel is written in
terms of wrapped Shen objects instead of native F# types.

[dr.mt...@gmail.com]

The trick there is that it's barely a compiler: ML-family languages
are actually fairly straightforward to build symbolic interpreters in,
and Shen does all the type-checking so you only have to worry about
KLambda. 

Ah, now it makes more sense.   If you're running an interpreter then

I can see how this would allow you to get a quart into a pint pot.  

Still not easy to implement though.  But you would lose performance

I'd guess.   Bruno's port is the most performant of them all so you might

find a time comparison interesting.  His Scheme port beats my SBCL

port by a factor of 4.

I more or less dumped the KL files and the Shen files w.o. giving 

instructions on how to assemble them.  In part because I figured that

each port might have its own language-specific way of assembly

and I'm not on top of 14 languages!    I concentrate mainly on making

sure what is there works.  Its good that Bruno is there to lend a hand.

Mark

[dr.mt...@gmail.com]

Apart from mastering F# and Shen - no small matter -

it seems from what I'm reading that Robert's original work

might have fallen victim to changing standards inside the

F# community - the so called 'bit rot' that often afflicts legacy

programs.  This can take a lot of tracking down.  I'm seeing now

why this has taken the time it has.  

M.

[Bruno Deferrari]

On Wed, May 15, 2024 at 5:31 PM dr.mt...@gmail.com <dr.mt...@gmail.com> wrote:

Keep in mind that Common Lisp is very different from other environments. Being able to just load the code and dump an image simplifies the work quite a bit. 

Quite true.  But there is one thing about Shen is that the kernel is very small.

For example,  under SBCL installing the kernel and the standard library takes

3 seconds.  In principle I could just skip saving an image and have a startup

that installs everything.  Maybe in other languages this is not true.  Anyway

I'm not going to second guess here because I've never used F#.

That is true, but there is another very important difference: in SBCL you still get fast object code even when evaluating KLambda/Lisp code at runtime. 

In contrast, doing this in Shen/Scheme would significantly hurt performance (precompiled code is faster than code that goes through "eval" at runtime). 

The impact is even greater in more statically typed languages like C, OCaml, and Rust. A Shen port in these languages could generate native code from KLambda in batch-compilation mode while using an interpreter for "eval." This would allow them to compile all standard Shen code, the standard library, and even user programs ahead of time for performance gains, while still supporting the REPL and "eval".

 

One thing I'm impressed with is that  Shen has been implemented in languages

like F# and Haskell.  I thought when I wrote Shen that it would only be supported

by languages that were dynamically typed since getting it into a Hindley-Milner

language would be akin to fitting a quart into a pint pot.  But people have done

what I thought would have been impossible.  Kudos.

M.

--

You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/1004d945-44a1-4f86-94d5-65848ea430efn%40googlegroups.com.

--

BD

[Jacob MacDonald]

Dr. Mark Tarver wrote:
> Bruno's port is the most performant of them all so you might
> find a time comparison interesting. His Scheme port beats my SBCL
> port by a factor of 4.

I sure intend to use it as an example for writing faster compiler
code. I also know that Chez is capable of absolutely cooking, whether
it's magic in the implementation or the wizards who use it. Sometimes
I'm sad that I developed more of a taste for Common Lisp, as I don't
have the juice to learn n Schemes. Guile being the one next on my
list.

> I more or less dumped the KL files and the Shen files w.o. giving
> instructions on how to assemble them. In part because I figured that
> each port might have its own language-specific way of assembly
> and I'm not on top of 14 languages!

Speaking of finding scaffolds (like shen.initialise) helpful, I'm very
excited about using Yggdrasil to "close the loop". That is, the
Yggdrasil docs describe the set of holes an implementation needs to
fill to provide a compilation target. I suspect that if I do that
work, the ability to develop large parts of ShenSharp in Shen will not
be far away.

[Bruno Deferrari]

On Wed, May 15, 2024 at 9:14 PM Jacob MacDonald <jacc...@gmail.com> wrote:

I sure intend to use it as an example for writing faster compiler
code. I also know that Chez is capable of absolutely cooking, whether
it's magic in the implementation or the wizards who use it.

Chez will get you good performance even if you don't try, but to get the best of it you have to help it:

https://cisco.github.io/ChezScheme/csug10.0/use.html#./use:h6

A Shen dialect without `eval` would let you produce faster programs (you will see that many of the modifications in the GitHub version

of the Shen kernel are about being able to avoid `eval` at the core).

But Shen/Scheme needs to support the Shen REPL, function redefinitions, etc, so it doesn't exploit Chez at its full potential. Still, given the constraints, I am pretty happy with the results.

A mode could be added to compile programs without `eval` support to produce faster executables, but I never got to it.

 

--

BD