Assistance with the shen-cl interop. Getting "function undefined".

[emum]

Hi, I downloaded shen-cl and looking at the interop.

I'm running (load "/home/path/to/interoptest/shen-cl-master/src/package.lsp") and can see that shen and shen-utils are registered

* (list-all-packages)
(#<PACKAGE "COMMON-LISP"> #<PACKAGE "SB-THREAD"> #<PACKAGE "SB-X86-64-ASM">
 #<PACKAGE "SB-C"> #<PACKAGE "SB-WALKER"> #<PACKAGE "SB-KERNEL">
 #<PACKAGE "SB-FASL"> #<PACKAGE "SB-IMPL"> #<PACKAGE "SB-UNIX">
 #<PACKAGE "SHEN"> #<PACKAGE "SB-GRAY"> #<PACKAGE "SB-PROFILE">
 #<PACKAGE "SB-PRETTY"> #<PACKAGE "SB-ALIEN"> #<PACKAGE "SB-APROF">
 #<PACKAGE "SHEN-UTILS"> #<PACKAGE "SB-SEQUENCE"> #<PACKAGE "SB-RBTREE">
 #<PACKAGE "SB-ASSEM"> #<PACKAGE "COMMON-LISP-USER"> #<PACKAGE "SB-DISASSEM">
 #<PACKAGE "SB-EVAL"> #<PACKAGE "SB-PCL"> #<PACKAGE "SB-DEBUG">
 #<PACKAGE "SB-MOP"> #<PACKAGE "SB-BIGNUM"> #<PACKAGE "SB-LOCKLESS">
 #<PACKAGE "SB-FORMAT"> #<PACKAGE "SB-VM"> #<PACKAGE "SB-LOOP">
 #<PACKAGE "SB-REGALLOC"> #<PACKAGE "SB-INT"> #<PACKAGE "SB-EXT">
 #<PACKAGE "SB-UNICODE"> #<PACKAGE "KEYWORD"> #<PACKAGE "SB-ALIEN-INTERNALS">
 #<PACKAGE "SB-DI"> #<PACKAGE "SB-SYS">)

Then I run (use-package :shen :shen-utils) and get T.

However when I run (shen-utils:load-shen "/home/path/to/interoptest/test.shen") I get undefined function: SHEN-UTILS:LOAD-SHEN.

Could use some advice why this is failing?

[Bruno Deferrari]

I don't know exactly how packages work in Common Lisp so this may be wrong, but I think that what you are doing there is just loading the package definitions but none of the Shen code, and that is why you get that undefined function error (the function has not been defined, just the package).

As a quick hacky solution:

- Download shen-cl-v3.0.3-sources.tar.gz, uncompress it and cd into that directory

- Run: make fetch

- Copy boot.lsp to shen-env.lsp and delete this part of the code in shen-env.lsp: https://github.com/Shen-Language/shen-cl/blob/c26776641fe4b63aacfbc6013fad9c1c16974a5e/boot.lsp#L176-L212
- Run: sbcl --load shen-env.lsp

After loading that file `load-shen` should work:

* (shen-utils:load-shen "/tmp/test.shen")
Test
"Test
"

run time: 4.699230194091797e-4 secs
SHEN::|loaded|

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/66b8f1e2-56e9-4a95-ac51-736750c3e77cn%40googlegroups.com.

--

BD

[emum]

Hi Bruno, thanks.

Followed those steps but when I load shen-env.lsp I get

"Directory "./compiled/" not found.

Run 'make precompile' to precompile the kernel and compiler."

Assuming this is referring to line #33 -- (defconstant compiled-path "./compiled/")

Should I run "make precompile" first or is there another recommendation? 

[emum]

NVM! Apologies. Figured that out.

[emum]

Ok but now I have test.shen with

(define shen-hello

Msg -> pr Msg)

and test.lsp with

(shen-utils:load-shen "/home/path/to/brunotest/shen-cl-v3.0.3-sources/test.shen")

(defun CL-HELLO (msg)

(shen:shen-hello msg))

I run "sbcl --load shen-env.lsp" and then (load "test.lsp") and I get

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {1000560083}>:
  syntax error in shen-hello here:

 Msg

Anything else I need to do to get the interop to work?

[emum]

ah jeez. Mistakes were made... that error was because my syntax was wrong.

test.shen is now

(define shen-hello

Msg -> (pr Msg))

However now when I load test.lsp I get

(load "test.lsp")
shen-hello
run time: 0.001000046730041504 secs
debugger invoked on a SB-C::INPUT-ERROR-IN-LOAD in thread

#<THREAD "main thread" RUNNING {1000560083}>:

  READ error during LOAD:
    Symbol "SHEN-HELLO" not found in the SHEN package.

[emum]

So to recap where I'm stuck, after following your advice above, Bruno

\\ test.shen

(define shen-hello

Msg -> (pr Msg))

;; test.lsp

(shen-utils:load-shen "/path/to/shen-cl-v3.0.3-sources/test.shen")

(defun CL-HELLO (msg)

(shen:shen-hello msg))

load-shen seems to work but then my cl-hello function causes error "Symbol "SHEN-HELLO" not found in the SHEN package."

Is this a bug in the load-shen function or am I not doing something?

Would love some guidance on this. Thanks.

[Bruno Deferrari]

Try this:

(defun CL-HELLO (msg) (shen::|shen-hello| msg))

Remember that Shen code is lowercased, while Common Lisp code is UPCASED (although by default you write in lowercase and the reader converts it).

By wrapping the symbol in |...| you force the Common Lisp reader to respect the casing typed by the user.

--

You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/91d3fb30-9878-4a3c-85c1-0ebbe50fbf3dn%40googlegroups.com.

--

BD

[emum]

BOOOOOOM works thanks Bruno!!

[emum]

So now last thing if I may, just to wrap this up:

Thus far I have been using the compiled environment from shen-cl-v3.0.3-sources.tar.gz as recommended, but I see that this was last updated Dec 07, 2019 per the timestamp on the releases page. Does that mean it does not get updated with the latest releases we get from the Download page such as S32.03?

Are we meant to consolidate the environment downloaded from the github repo and Downloads page? What is the process for that?

[emum]

In other words, I see that when you run "make fetch" the kernel/ directory is generated.

Is it just a matter of migrating the kernel files you get from the Downloads page into this directory? Like overwrite sources/ files with Sources/ from Downloads, overwrite klambda/ with Klambda/ files, and so forth?

Do I need to do this manually or do we have any tooling that does this for you? Or what is the right way of using the latest shen version?

Let me know if the question makes sense. Thanks.

[Bruno Deferrari]

The Common Lisp port available on GitHub is not yet compatible with the S-series kernel.

Replacing the kernel files is not enough because the new kernels and the old compiler are incompatible.

Most of the work has already been done when upgrading Shen/Scheme so that it is compatible with the S-series kernel (and also when releasing the adapted kernel version that can be found on GitHub), but for the Common Lisp port to work there is still some work required (but I have no estimate for this).

A secondary issue is that with that with the shutdown of travis-ci.org (see last point here https://en.wikipedia.org/wiki/Travis_CI#Company) there is no system in place for releasing binaries for that port right now (needs to be rewritten to use GitHub Actions like with Shen/Scheme releases), although for you this probably wouldn't be an issue since you are compiling from source code anyway.

--

You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/16a14384-4a13-47eb-ab99-32bd820dc7e5n%40googlegroups.com.

--

BD

[emum]

Ah damn :( Alright thank you for explaining that. The shen book uses the CL port though?

When you say you don't have an estimate for implementing that compatibility, do you mean like none at all or just nothing too precise.

Can you tell me if we're talking on the order of a year or two? Or is this part of the project really on hold indefinitely.

I'll check out scheme.

[emum]

Alternatively, could you tell me if Shen S-series will be implementing web support anytime soon, for example to create a web server with? That's really the only reason I need any of this.

[emum]

Bruno, for this final step you provided above "- Run: sbcl --load shen-env.lsp", what is the equivalent for shen-scheme please? How do you load your scm files such that they're ready for interop with shen?

I'm looking at the "Running" section of the readme, but that only talks about the shen-scheme binary which only loads shen files, correct? Then I thought maybe you're supposed to run src/init.scm but that gives me "Exception: variable kl:set is not bound".

I'm not seeing a boot.scm file like there is the boot.lsp for the cl port. Do I need to run .../shen-scheme-v0.25.3-src/_build/lib/shen-scheme/shen.boot ? If so, can you tell me how to actually run it? I'm googling how to run .boot file extensions on macos but not finding anything.

Otherwise, what do I run? -- Thanks

[Bruno Deferrari]

No, much less than that. It is not so much about the amount of work but about finding the time and focus to do it (all the hardest parts were already solved when upgrading Shen/Scheme, since Shen/CL shares most of the compiler code with Shen/Scheme, it is about repeating the process but for Shen/CL).

 

I'll check out scheme.

--

You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/39b0c8f0-6d54-4c71-b3e1-b38cadae41aen%40googlegroups.com.

--

BD

[Bruno Deferrari]

In Shen/Scheme it would be a bit different. I will add an easier way to do it when I release a new version, but for now try this:

- After compiling, copy shen-scheme.scm to load-shen.scm

- Comment out or delete the line that starts with "(define-top-level-value 'get-shen-scheme-home-path ..."

- Comment out and delete the expression at the end of the file that starts with "(scheme-start ..."

- Add this at the end of the file: (initialize-shen)

Then from the Chez REPL you can do:

Chez Scheme Version 9.5.4
Copyright 1984-2020 Cisco Systems, Inc.

> (load "load-shen.scm")
> (kl:load "/tmp/test.shen") ; file contains (define test X Y -> (+ X Y))
(fn test)

run time: 0.0015089999999999826 secs
loaded
> (kl:test 1 2)
3

All the Shen functions are prefixed with "kl:" in Scheme, so "kl:load" is calling Shen's "load" function. "kl:eval" will call Shen's `eval` function, etc:

> (load "load-shen.scm")
> (kl:eval '(+ 1 2 3))
6
> (kl:eval '(define test X Y -> (+ X Y)))
#(shen.printF "(fn test)")
> (kl:test 4 5)
9

 

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/qilang/034de024-843d-45b0-809b-0bffebeb3b9bn%40googlegroups.com.

--

BD

[emum]

Brilliant Bruno, thank you! 🔥🔥🔥 Saw your response on the other threat as well. This all works. Thanks!

[emum]

"Thread", not "threat" :)