Upgrading port to S series: Final bits.

[Jacob MacDonald]

My punctuated attempt at updating ShenSharp continues.

I'm running into a few last speed bumps and cannot find documentation
on them. I'm using the 38.1 release as provided by Bruno on GitHub;
Eventually I'll look to copy the 38.2 package from the site.

There are a few functions that I do not have definitions for. That's
fine, I can implement them in the runtime, but they seem to be extant
in the declarations but only used during initialisation now. The
culprits:

- interror
- list
- shen.pprint

Are there definitions or specifications for these out there somewhere?

In addition, ShenSharp thinks that foreign is a function it needs to
have, but as I understand it that form is something like a tag and we
can simply unwrap the form during read/deal with CLR calls as
appropriate. Am I on the right track?

I encountered these functions bringing init.kl back into the build. Is
that still a necessary step? Feels like it given all the runtime
information unavailable without it, but I can't quite tell what the
process from Shen to KL to a port looks like these days.

Thanks!

Jacob.

[Bruno Deferrari]

Hi Jacob, 38.2 is on GitHub now.

On Sun, May 12, 2024 at 9:07 AM Jacob MacDonald <jacc...@gmail.com> wrote:

My punctuated attempt at updating ShenSharp continues.

I'm running into a few last speed bumps and cannot find documentation
on them. I'm using the 38.1 release as provided by Bruno on GitHub;
Eventually I'll look to copy the 38.2 package from the site.

There are a few functions that I do not have definitions for. That's
fine, I can implement them in the runtime, but they seem to be extant
in the declarations but only used during initialisation now. The
culprits:

- interror
- list
- shen.pprint

Are there definitions or specifications for these out there somewhere?

Hmmm, I think `interror` used to exist long ago, not sure about the others. What I see is that those names are still referenced when building the arity table, so I guess that is what is causing the issue. For now you can try just ignoring them or removing them from the list in declarations.shen (or directly in init.kl if you are not going to regenerate the klambda files).

 

In addition, ShenSharp thinks that foreign is a function it needs to
have, but as I understand it that form is something like a tag and we
can simply unwrap the form during read/deal with CLR calls as
appropriate. Am I on the right track?

It is a special form, it probably didn't exist when the last version of ShenSharp was written. But It gets processed at the Shen compiler level, you shouldn't be seeing `foreign` when you are processing KLambda.

 

I encountered these functions bringing init.kl back into the build. Is
that still a necessary step? Feels like it given all the runtime
information unavailable without it, but I can't quite tell what the
process from Shen to KL to a port looks like these days.

If ShenSharp made use of it in previous versions you still need it, otherwise you will need to do all the evaluation of the initialization code at runtime. What init.kl gives you is an entry-point function to the initialization code that you can compile ahead of time (it sounds like extra work but it actually simplifies things for ports and helps with performance).

Re: process for ports, it is probably simpler now because there are fewer things to worry about (much of what the port had to do for handling functions is now handled by the Shen->Kl compiler for example). But for ShenSharp it is likely that it already contains some of the logic that is not needed anymore, along with optimizations that make assumptions that are not true anymore.

Thanks!

Jacob.

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/CACy6W0AiiES030RvQ-e%3DR8Gc3pyuYLnkAhMuaNO%2BYdQikOA5mg%40mail.gmail.com.

--

BD

[dr.mt...@gmail.com]

It looks like shen.pprint and interror left the scene

some time back but were left in the declarations file

I'd ignore those entries.  I'll remove them in the next release.

'list' is a type operator; not a function.  'foreign'

is indeed a tag.

I know nothing of init.kl.   I have no such file in my distribution.

Mark

[Bruno Deferrari]

On Sun, May 12, 2024 at 9:34 AM dr.mt...@gmail.com <dr.mt...@gmail.com> wrote:

It looks like shen.pprint and interror left the scene

some time back but were left in the declarations file

I'd ignore those entries.  I'll remove them in the next release.

'list' is a type operator; not a function.  'foreign'

is indeed a tag.

I know nothing of init.kl.   I have no such file in my distribution.

init.kl exposes one function `shen.initialise`, which when called executes all the code that initializes the runtime state (arity table, lambda forms table, function signatures, etc).

This simplifies the process for platforms that don't offer the option to dump a runtime image that can be loaded later.

 

Mark

On Sunday 12 May 2024 at 13:07:53 UTC+1 jacc...@gmail.com wrote:

My punctuated attempt at updating ShenSharp continues.

I'm running into a few last speed bumps and cannot find documentation
on them. I'm using the 38.1 release as provided by Bruno on GitHub;
Eventually I'll look to copy the 38.2 package from the site.

There are a few functions that I do not have definitions for. That's
fine, I can implement them in the runtime, but they seem to be extant
in the declarations but only used during initialisation now. The
culprits:

- interror
- list
- shen.pprint

Are there definitions or specifications for these out there somewhere?

In addition, ShenSharp thinks that foreign is a function it needs to
have, but as I understand it that form is something like a tag and we
can simply unwrap the form during read/deal with CLR calls as
appropriate. Am I on the right track?

I encountered these functions bringing init.kl back into the build. Is
that still a necessary step? Feels like it given all the runtime
information unavailable without it, but I can't quite tell what the
process from Shen to KL to a port looks like these days.

Thanks!

Jacob.

--

You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/aa7b8620-79f5-4173-9e4f-e0f002379815n%40googlegroups.com.

--

BD

[Jacob MacDonald]

Thanks for the help. I ended up hacking on the shenlanguage.org
distributions after sending that email, which leads me to my first
actionable question and the rest of the below.

Is there a way to pretty-print KLambda? Debugging init.kl with its
long long lines proves difficult.

My second question was how that file is generated, but I had simply
skipped over the relevant Makefile rules. See the very end of this
email, though.

Bruno Deferrari wrote:
> What I see is that [interror, ...] are still referenced when building the arity table, so I guess that is what is causing the issue.

The arity table is what I found first, but I think it's not exactly
the case. See

> you shouldn't be seeing `foreign` when you are processing KLambda.

I see this form in init.kl (38.2 version from GitHub now)

(shen.set-lambda-form-entry (cons foreign (lambda Y1096 (foreign Y1096))))

As far as I can tell the earlier instances of "foreign" are symbols in
cons cells in the arity table. This is what I think is getting picked
up as an application. There are similar instances of the other problem
symbols below that block of consing (which I believe is the arity
table).

> If ShenSharp made use of it in previous versions you still need it, otherwise you will need to do all the evaluation of the initialization code at runtime. What init.kl gives you is an entry-point function to the initialization code that you can compile ahead of time (it sounds like extra work but it actually simplifies things for ports and helps with performance).

> ...

> Re: process for ports, it is probably simpler now because there are fewer things to worry about (much of what the port had to do for handling functions is now handled by the Shen->Kl compiler for example). But for ShenSharp it is likely that it already contains some of the logic that is not needed anymore, along with optimizations that make assumptions that are not true anymore.

I'm curious about simplification and performance, but that feels like
a fork of this thread. For now, I'll lay out how ShenSharp builds
itself as I understand it. A compiler translates KLambda to F# code,
which becomes part of a runtime assembly which is capable of running
all of Shen. It does depend on having shen.initialise available, and
thus skips any toplevel forms other than defuns. I want to refactor
this to record all toplevel forms, even the ones we don't currently
compile. Without fixing that, I can get a half-working REPL by
hardcoding the skipped (set ...)s into the runtime. So I'm moving down
the pathway of using the official website's package and eliminating
the hack.

Finally, I'll revisit my pseudo-question about the init.kl build from
above. shen.initialise throws the wrenches already mentioned, but it
also requires the <-dict and dict-> functions, which ShenSharp got
from dict.kl before. That's another file which exists only in the
GitHub distribution.

Jacob.

[Jacob MacDonald]

Jacob MacDonald wrote:
> Is there a way to pretty-print KLambda? Debugging init.kl with its
> long long lines proves difficult.

As is tradition, immediately on sending the message I proceeded to the
logical next step. With a good-enough pprint, I can see that the
seemingly problematic applications are in
shen.initialise-lambda-forms. Shen 22.4, the last version that fully
works with my ShenSharp fork, obviously doesn't contain foreign, but
it does have interror in shen.initialise_arity_table, but not in
initialise-lambda-forms. 38.1 contains interror in the latter, which
is one place my compiler chokes.

Jacob.

[Jacob MacDonald]

Oh, and a final little note not documented yet in my port attempt: The
compiler for ShenSharp doesn't handle redefinitions in KL, and there
are two that I manually patch out of the official distribution so that
it builds. They seem syntactically identical though the gensyms in the
KL versions differ.

internal-to-shen? at two places in reader
(https://github.com/Shen-Language/shen-sources/blob/shen-38.2/sources/reader.shen#L481
and https://github.com/Shen-Language/shen-sources/blob/shen-38.2/sources/reader.shen#L559)
semicolon? in prolog
(https://github.com/Shen-Language/shen-sources/blob/shen-38.2/sources/prolog.shen#L158)
and yacc (https://github.com/Shen-Language/shen-sources/blob/shen-38.2/sources/yacc.shen#L74)

I delete the latter definition based on file load order.

[Bruno Deferrari]

On Sun, May 12, 2024 at 6:47 PM Jacob MacDonald <jacc...@gmail.com> wrote:

Thanks for the help. I ended up hacking on the shenlanguage.org
distributions after sending that email, which leads me to my first
actionable question and the rest of the below.

Is there a way to pretty-print KLambda? Debugging init.kl with its
long long lines proves difficult.

My second question was how that file is generated, but I had simply
skipped over the relevant Makefile rules. See the very end of this
email, though.

Hi Jacob, please check https://github.com/Shen-Language/shen-sources/blob/master/extensions/expand-dynamic.shen

What it does is what you describe at the end of your email. It collects everything that is not a defun, and bundles it inside

initialization functions that you can call during startup (with `shen.initialize` calling everything in order).

 

Bruno Deferrari wrote:
> What I see is that [interror, ...] are still referenced when building the arity table, so I guess that is what is causing the issue.

The arity table is what I found first, but I think it's not exactly
the case. See

> you shouldn't be seeing `foreign` when you are processing KLambda.

I see this form in init.kl (38.2 version from GitHub now)

    (shen.set-lambda-form-entry (cons foreign (lambda Y1096 (foreign Y1096))))

As far as I can tell the earlier instances of "foreign" are symbols in
cons cells in the arity table. This is what I think is getting picked
up as an application. There are similar instances of the other problem
symbols below that block of consing (which I believe is the arity
table).

I think the issue is that in declarations.shen, this is called:

(build-lambda-table (external shen))

 

which will, for each external Shen symbol insert an entry in the lambda table (/. X (external-symbol X)), but not every external symbol is a function.

But that lambda function itself is just constructed, and not called. While it is not ideal that it references a function that doesn't exist, that is still something that needs to be supported in a full implementation (you can define functions that reference other functions that have not been defined yet in Shen and Klambda, you will get an error at runtime if you try to call that function, but it should still compile).

Btw, you see this in init.kl, but it is not the source of the issue. You would have to still execute this code somehow to recover the runtime state required by Shen, init.kl is just a dump of the code that performs that.

> If ShenSharp made use of it in previous versions you still need it, otherwise you will need to do all the evaluation of the initialization code at runtime. What init.kl gives you is an entry-point function to the initialization code that you can compile ahead of time (it sounds like extra work but it actually simplifies things for ports and helps with performance).
> ...
> Re: process for ports, it is probably simpler now because there are fewer things to worry about (much of what the port had to do for handling functions is now handled by the Shen->Kl compiler for example). But for ShenSharp it is likely that it already contains some of the logic that is not needed anymore, along with optimizations that make assumptions that are not true anymore.

I'm curious about simplification and performance, but that feels like
a fork of this thread. For now, I'll lay out how ShenSharp builds
itself as I understand it. A compiler translates KLambda to F# code,
which becomes part of a runtime assembly which is capable of running
all of Shen. It does depend on having shen.initialise available, and
thus skips any toplevel forms other than defuns. I want to refactor
this to record all toplevel forms, even the ones we don't currently
compile. Without fixing that, I can get a half-working REPL by
hardcoding the skipped (set ...)s into the runtime. So I'm moving down
the pathway of using the official website's package and eliminating
the hack.

The simplification is that it already does what you want to do here. The performance advantage comes from the fact that this setup allows you to compile everything ahead of time (to the target bytecode or language) and allows you to start the system without a single call to `eval`.

 

Finally, I'll revisit my pseudo-question about the init.kl build from
above. shen.initialise throws the wrenches already mentioned, but it
also requires the <-dict and dict-> functions, which ShenSharp got
from dict.kl before. That's another file which exists only in the
GitHub distribution.

Yes, same as init.kl, dict.kl is not part of the original distribution. What it does is to allow the underlying platform to use its own hash tables and hashing primitives which are usually much faster than the default portable ones.

 

Jacob.

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/CACy6W0BkKXmghtmYjhYBy9wZvtQg2n3CPvrhVgvVwwx8ScRGNg%40mail.gmail.com.

--

BD

[Bruno Deferrari]

Good catch.

 

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/CACy6W0AV1V-9M7JGgEjKGo1g%2BZ3wc8zKY9fgBVoZK65Zg8%3DuwA%40mail.gmail.com.

--

BD

[Jacob MacDonald]

Bruno Deferrari wrote:
> Yes, same as init.kl, dict.kl is not part of the original distribution. What it does is to allow the underlying platform to use its own hash tables and hashing primitives which are usually much faster than the default portable ones.

I was under the impression that the dict-function situation was the
opposite: dict.shen contains the portable implementation which can be
shadowed. In any case, what's relevant for porting is not dict.shen
but the fact that the website's .kl files don't use dict functions
while the ones on GitHub do. Where do they get inserted from?

> While it is not ideal that it references a function that doesn't exist, that is still something that needs to be supported in a full implementation

> ...

> Btw, you see this in init.kl, but it is not the source of the issue. You would have to still execute this code somehow to recover the runtime state required by Shen, init.kl is just a dump of the code that performs that.

OK, I can't claim to understand it fully without hacking on the
underlying Shen code, I get the gist. However, what has happened is
that this issue did not crop up in the 22 kernel but does in 38. I
could bisect to see exactly where the regression is, but since
interror exists in the old initialiser's arity collection, the
breakage I'm seeing has *something* to do with changes to the rest of
the initialiser.

> The simplification is that it already does what you want to do here. The performance advantage comes from the fact that this setup allows you to compile everything ahead of time (to the target bytecode or language) and allows you to start the system without a single call to `eval`.

Sounds like the ShenSharp compiler targeting 22.4 didn't actually
target a sufficient subset of KLambda, then? What I'm hearing is that
the initialisation function does some compilation lifting into
KLambda, but while an older version of ShenSharp used it and passed
tests, it can no longer build its F# output successfully.

Jacob.

[Bruno Deferrari]

On Sun, May 12, 2024 at 7:47 PM Jacob MacDonald <jacc...@gmail.com> wrote:

Bruno Deferrari wrote:
> Yes, same as init.kl, dict.kl is not part of the original distribution. What it does is to allow the underlying platform to use its own hash tables and hashing primitives which are usually much faster than the default portable ones.

I was under the impression that the dict-function situation was the
opposite: dict.shen contains the portable implementation which can be
shadowed. In any case, what's relevant for porting is not dict.shen
but the fact that the website's .kl files don't use dict functions
while the ones on GitHub do. Where do they get inserted from?

Correct. The original distribution contains and uses this functionality, but it is not organized as a "dictionary" API. With dict.kl you have it reorganized in a way that exposes a dictionary API with explicit names that you can shadow in your port to use instead faster, native versions. See https://github.com/Shen-Language/shen-sources/blob/master/doc/port-upgrades.md#dictionaries

(also, that file in general could be useful for you to read to understand how things went up until the point the S kernel got released, that doc hasn't been updated since)

 

> While it is not ideal that it references a function that doesn't exist, that is still something that needs to be supported in a full implementation
> ...
> Btw, you see this in init.kl, but it is not the source of the issue. You would have to still execute this code somehow to recover the runtime state required by Shen, init.kl is just a dump of the code that performs that.

OK, I can't claim to understand it fully without hacking on the
underlying Shen code, I get the gist. However, what has happened is
that this issue did not crop up in the 22 kernel but does in 38. I
could bisect to see exactly where the regression is, but since
interror exists in the old initialiser's arity collection, the
breakage I'm seeing has *something* to do with changes to the rest of
the initialiser.

Do you see any special-casing for interror somewhere else in the ShenSharp compiler? Maybe there was some code specific to it, "foreign" is a new symbol.

 

> The simplification is that it already does what you want to do here. The performance advantage comes from the fact that this setup allows you to compile everything ahead of time (to the target bytecode or language) and allows you to start the system without a single call to `eval`.

Sounds like the ShenSharp compiler targeting 22.4 didn't actually
target a sufficient subset of KLambda, then?

It could be, or maybe it didn't handle it at compilation time but worked in the REPL (I am not familiar with the internals of ShenSharp, but I know that for Shen/Scheme I expect a stricter subset of Shen when compiling the core than when running user programs, otherwise some things would break, and others would be slower).

 

What I'm hearing is that
the initialisation function does some compilation lifting into
KLambda, but while an older version of ShenSharp used it and passed
tests, it can no longer build its F# output successfully. 

Jacob.

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/CACy6W0DAWRjg1CG_BzJz21PmpQc7evP5%2Bq8Kt7w3%2BY9_Ng_G_g%40mail.gmail.com.

--

BD

[Jacob MacDonald]

Bruno Deferrari wrote:
> Do you see any special-casing for interror somewhere else in the ShenSharp compiler? Maybe there was some code specific to it, "foreign" is a new symbol.

No, AFAICT the only special-case in the compiler is for cd, which has
a fully F# implementation. Well, that and the skipping of non-defuns.
The new issue with interror/foreign/etc. is that they appear in
initialise-lambda-forms and not only as data in the arity forms. I
admit that I'm a bit thrown by the differences between make.shen on
GitHub and make.shen in the tarball from GitHub. "lambda-forms" does
not appear in the tarball except in init.kl, as far as I can find. And
while make.shen (tarball, not GitHub) references the function
(shen.initialise-lambda-forms), it doesn't contain the definition.

It seems to me that there are several interacting runtimes and I don't
understand their relationship.

I want to create a F#/Shen 38 runtime. To do so I need KL files
generated from/reflecting a working Shen runtime. The ones on the Shen
site are from the bundled CL implementation and the ones on GitHub are
generated by a Scheme implementation. Is that correct?

My instinct is to choose a source of truth to branch the F# runtime
from, but the docs on porting are slightly unclear. I expect that the
KL files from the website are sufficient for a working environment if
I implement non-defun compilation, but init.kl, which does that
lifting for me, includes KL generated from somewhere unclear and
breaks the ShenSharp compiler.

Jacob.

[Bruno Deferrari]

Just checked and in version 22 interror was not an external symbol. It had an arity defined but that was it.

The problem you are seeing happens when these things happen at the same time:

- A symbol is defined as external to the shen package.

- It has an arity defined.

- No function with that name exists.

The Shen kernel will try to create a lambda form for each symbol that fulfills the first two conditions.

As a quick workaround you can define dummy functions with those names (and arities). So inside any of those .kl files you can add: 

(defun foreign X skip)

(defun interror X Y skip)

...etc

But in general, unless there is a reason I am unaware of for these to have an arity, we can remove them from the table in the Shen sources.

Mark, would that cause any issue? (I don't know if there is external code that depends on that or these are just leftovers)

On Sun, May 12, 2024 at 8:46 PM Jacob MacDonald <jacc...@gmail.com> wrote:

Bruno Deferrari wrote:
> Do you see any special-casing for interror somewhere else in the ShenSharp compiler? Maybe there was some code specific to it, "foreign" is a new symbol.

No, AFAICT the only special-case in the compiler is for cd, which has
a fully F# implementation. Well, that and the skipping of non-defuns.
The new issue with interror/foreign/etc. is that they appear in
initialise-lambda-forms and not only as data in the arity forms. I
admit that I'm a bit thrown by the differences between make.shen on
GitHub and make.shen in the tarball from GitHub. "lambda-forms" does
not appear in the tarball except in init.kl, as far as I can find. And
while make.shen (tarball, not GitHub) references the function
(shen.initialise-lambda-forms), it doesn't contain the definition.

You will not find such definition in the original sources because the processing to collect all non-defun code and put it inside initialization functions has not been made there.

 

It seems to me that there are several interacting runtimes and I don't
understand their relationship.

I want to create a F#/Shen 38 runtime. To do so I need KL files
generated from/reflecting a working Shen runtime. The ones on the Shen
site are from the bundled CL implementation and the ones on GitHub are
generated by a Scheme implementation. Is that correct?

Yes, but any working and current Shen implementation will produce the same code (the code producing that output is written in Shen).

Then each port can then further post-process that code depending on its needs (Shen/Scheme then performs an extra pass to override some functions).

 

My instinct is to choose a source of truth to branch the F# runtime
from, but the docs on porting are slightly unclear. I expect that the
KL files from the website are sufficient for a working environment if
I implement non-defun compilation, but init.kl, which does that
lifting for me, includes KL generated from somewhere unclear and
breaks the ShenSharp compiler.

Jacob.

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/CACy6W0DxONd7ovhBAi6h_FLVoR9jDk-g3xNWkoB%3D%2B49znY0rHQ%40mail.gmail.com.

--

BD

[Jacob MacDonald]

Jacob MacDonald wrote:
> I admit that I'm a bit thrown by the differences between make.shen on
> GitHub and make.shen in the tarball from GitHub. "lambda-forms" does
> not appear in the tarball except in init.kl, as far as I can find. And
> while make.shen (tarball, not GitHub) references the function
> (shen.initialise-lambda-forms), it doesn't contain the definition.

Silly misreading by me. I'm still curious about the relationships of
the sources to one another and the best approach to porting, but it is
entirely possible that the KL->F# compiler is not sufficient for the
new version of shen.initialise, and I can see where to read that
function in Shen and not KL now. ShenSharp's previous approach was to
build a file containing a series of installDefun expressions which
assigned function objects to symbol slots, then a call to the
generated F# shen.initialise. If initialise-lambda-forms is roughly
equivalent to installDefun, I can easily see one being too eager in
cases like interror or foreign. Was that introduced as a new feature
of initialise?

Jacob.

[Bruno Deferrari]

Try this version of init.kl, it was regenerated with the arity for those two symbols removed, and see if you find another symbol that has no associated function:

https://gist.githubusercontent.com/tizoc/4c856f9b5a518a93a7b3024123cdfa08/raw/b9dbf50623f1c994e872d5b19e6486f30b65814c/init.kl

No, it is just that now those symbols show up there (because of the reasons I mentioned in the previous email), which was not something that used to happen before.

 

Jacob.

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/CACy6W0Cn7wz2KejyP7WA6FPNvnv%2B3XKJEZ_TqDOJow1vhVdA_Q%40mail.gmail.com.

--

BD

[Jacob MacDonald]

Bruno Deferrari wrote:
> Try this version of init.kl, it was regenerated with the arity for those two symbols removed, and see if you find another symbol that has no associated function:

Wonderful, with that I can get 38.2 up and running after shadowing
list and shen.pprint, which were the other two failures.

> No, it is just that now those symbols show up there (because of the reasons I mentioned in the previous email), which was not something that used to happen before.

I appreciate the explanation; As you can tell I was on the completely
wrong track. Interestingly, the test suite fails when mutuals is
loaded, and I can observe this strange behavior in my REPL.

Shen, www.shenlanguage.org, copyright (C) 2010-2023, Mark Tarver
version: S38.2, language: F#, platform: dotnet 8.0.4
port 0.10.0.0, ported by Robert Koeninger


(0-) (define even*? 1 -> false X -> (odd*? (- X 1)))
(fn even*?)

(1-) (define odd*? 1 -> true X -> (even*? (- X 1)))
(fn odd*?)

(2-) (even*? 1)
false

(3-) (even*? 2)
fn: odd*? is undefined

(4-) (odd*? 1)
fn: odd*? is undefined

(5-) (exit 0)
fn: exit is undefined

(6-) %

Clearly I have some bugs to iron out.

Thanks,

Jacob.

[Jacob MacDonald]

The failing tests are another example of something that used to exist
as well, very slightly renamed (compare
https://github.com/Shen-Language/shen-sources/blob/shen-22.4/tests/tests.shen#L77
to https://github.com/Shen-Language/shen-sources/blob/shen-38.2/tests/kerneltests.shen#L35).
I was wrong already about the interaction of ShenSharp's compiler and
the new Shen runtime, so I'm not sure what would cause this
regression. But I've made a note of it and the odd behavior of exit in
the PR.

[Jacob MacDonald]

Jacob MacDonald wrote:
> Clearly I have some bugs to iron out.

They don't seem to be exactly about the mutuality, though. I assume
this is a bug in the F# code somewhere, but I'm putting it here unless
I'm missing something obvious.

Even a simple definition of odd*? fails in the same way:

(0-) (define odd*? X -> true)
(fn odd*?)

(1-) (fn odd*?)
fn: odd*? is undefined

(2-) (odd*?)
<Interpreted Function>

The problem isn't with the symbol itself: The function shows up in
partial form and I can set the symbol's value.

(3-) (set odd*? true)
true

(4-) (value odd*?)
true

even*? does not exhibit the same problem.

(0-) (define even*? X -> true)
(fn even*?)

(1-) (fn even*?)
<Interpreted Function>

(2-) (even*?)
<Interpreted Function>

[Jacob MacDonald]

I've tracked down some of the extant bugs to the S series improvements
to function definition, currying, etc. ShenSharp loads its builtin
functions into the global environment before running the generated
install step. It is unaware of shen.lambda-form attributes, though,
which means that only functions exported in init or with arity 0 work
(since the macro seems to simplify 0-ary calls even if the function
isn't known). This doesn't quite explain the difference between (fn
odd*?) and (fn even*?), but fixing (fn) gets me closer. What's the
right way to make sure that lambda-form attributes get set properly?
There was a leftover post-install step which, while not strictly
necessary, fixes some of our builtins by providing arity information
(which also bypasses the undefined function check).

[Jacob MacDonald]

In order to find the odd*?/even*? bug, I think I need a way to trace
my (define ...)s. Is there a way to do this?

I've broken the syntax down into F# calls, but I don't see a way to
compare the intermediate results before the functions are registered.

These ASTs are the output of process-applications
(https://github.com/Shen-Language/shen-sources/blob/shen-38.3/sources/sys.shen#L11)
and are identical as far as I can tell.

Cons
(Sym "define",
Cons (Sym "odd*?",
Cons (Sym "X", Cons (Sym "->", Cons (Sym "true", Empty)))))

Cons
(Sym "define",
Cons (Sym "even*?",
Cons (Sym "X", Cons (Sym "->", Cons (Sym "true", Empty)))))

However, if I extend the pre-REPL code to call shen->kl, I get
identical print statements but with one function successfully created
and the other not.

Vec [|Sym "shen.printF"; Str "(fn odd*?)"|]

Shen, www.shenlanguage.org, copyright (C) 2010-2023, Mark Tarver

version: S38.3, language: F#, platform: dotnet 8.0.4

port 0.10.0.0, ported by Robert Koeninger

(0-) (shen.<-dict (value *property-vector*) odd*?)
value odd*? not found in dict

Vec [|Sym "shen.printF"; Str "(fn even*?)"|]

Shen, www.shenlanguage.org, copyright (C) 2010-2023, Mark Tarver

version: S38.3, language: F#, platform: dotnet 8.0.4

port 0.10.0.0, ported by Robert Koeninger

(0-) (shen.<-dict (value *property-vector*) even*?)
[[arity | 1] [shen.lambda-form | <Interpreted Function>]
[shen.source defun even*? [V11] true]]