raco exe & distribute, and namespace-require'ing a module: missing racket/base?

[Jesse Alama]

I'm working on building a standalone executable for a #lang that can be used in two ways:

1. foo awesome.foo: execute file awesome.foo, which is written in #lang foo

2. foo (no arguments): fire up a REPL. Expressions are to be written in the foo language.

I can get (1) to work, after wrestling with raco exe (using ++lang) and raco distribute (using ++collects-copy). An invocation that works for me is:

### begin script snippet

raco exe ++lang foo ++lang racket/base ++lang brag foo.rkt

# copy the source code of foo into collects/foo

mkdir -p collects/foo

find . -mindepth 1 -maxdepth 1 -name '*.rkt' -exec cp {} collects/foo ';'

raco distribute ++collects-copy collects dist foo

### end script snippet

This works for making a standalone executable that can exectute foo programs specified on the command line, but doesn't work for a REPL. The difficulty seems to be the `namespace-require` part of `run-repl`, defined like this:

````

(define (run-repl)

  (parameterize ([current-namespace (make-base-empty-namespace)])

    (namespace-require 'foo/expander)

    (read-eval-print-loop)))

````

When I invoke the executable with no arguments, `run-repl` gets called. But this leads to:

````

standard-module-name-resolver: collection not found

  for module path: racket/base/lang/reader

  collection: "racket/base/lang"

````

expander.rkt in foo is written in racket/base, so I suppose that's what triggers the issue.

My question, crudely put, is: How do I "embed" racket/base so that I can use `namespace-require` as seen in `run-repl`? (I assume namespace-require is what I need; if that's wrong, please do correct me.) It appears that ++lang racket/base bit in raco exe isn't enough. A related question about raco exe is: if racket/base isn't available, what *is* available?

[Jesse Alama]

An update to my command line REPL adventures:

I'm still unable to get a working REPL, even when adding more and more information to raco exe/distribute. Here's my latest attempt:

raco exe ++lang foo ++lang racket/base ++lang racket ++lang brag ++lib racket ++lib racket/base foo.rkt

I've even taken to copying the entire Racket collects directory (!) based on the idea that the built-in executable is potentially looking for all sorts of things. Thus, after my raco exe and raco distribute invocations, I've got a build directory that looks like this:

$ ls /tmp/dist/lib/plt/foo/collects

acks

db

file

json

openssl

racket

realm

setup

xml

compiler

dynext

info

launcher

pkg

raco

foo # my package; everything else is standard Racket

syntax

data

ffi

info-domain

net

planet

reader

s-exp

version

In other words, I'm not only embedding a bunch of stuff into the generated executable, but making a ton of source code available, too. This all looks terribly excessive, and probably it is, but I've thrown all this stuff in out of desperation because I'm trying to ensure that racket/base can be used as lang via `namespace-require`.  Nonetheless, I still get a module-not-found error:

````

instantiate: unknown module

  module name: #<resolved-module-path:(submod '#%embedded:racket/base: reader)>

  context...:

   raise-arguments-error

   namespace-module-instantiate!96

   for-loop

   for-loop

   run-module-instance!125

   do-dynamic-require5

   read*14

   thunk_11

   dynamic-wind

   default-load-handler

   standard-module-name-resolver

   module-path-index-resolve5

   perform-require!78

   /var/folders/rx/lzgw7dzs0fqc1c7q1bv790_h0000gn/T/tmp.FrRgf7Mu/foo.rkt:64:0: run-repl

   call-with-values

   body of '|#%mzc:foo(main)|

````

Any idea what might be going on?

[Alex Harsanyi]

On Wednesday, September 25, 2019 at 1:37:41 PM UTC+8, Jesse Alama wrote:

I'm working on building a standalone executable for a #lang that can be used in two ways:

1. foo awesome.foo: execute file awesome.foo, which is written in #lang foo

2. foo (no arguments): fire up a REPL. Expressions are to be written in the foo language.

I can get (1) to work, after wrestling with raco exe (using ++lang) and raco distribute (using ++collects-copy). An invocation that works for me is:

### begin script snippet

raco exe ++lang foo ++lang racket/base ++lang brag foo.rkt

# copy the source code of foo into collects/foo

mkdir -p collects/foo

find . -mindepth 1 -maxdepth 1 -name '*.rkt' -exec cp {} collects/foo ';'

raco distribute ++collects-copy collects dist foo

### end script snippet

This works for making a standalone executable that can exectute foo programs specified on the command line, but doesn't work for a REPL. The difficulty seems to be the `namespace-require` part of `run-repl`, defined like this:

````

(define (run-repl)

  (parameterize ([current-namespace (make-base-empty-namespace)])

    (namespace-require 'foo/expander)

    (read-eval-print-loop)))

````

This is a wild guess, but since no one replied, I would suggest two things to try:

1) try using `make-base-namespace` instead -- I am not sure what the difference is between "attached" and "attached + required", but sounds like `make-base-namespace` will require racket/base

2) try a compiled form:

    (define (run-repl)

      (parameterize ([current-namespace (make-base-empty-namespace)])

        (compile `(namespace-require `'foo/expander))

        (read-eval-print-loop)))

Alex. 

[Matthew Flatt]

At Tue, 24 Sep 2019 22:37:41 -0700 (PDT), Jesse Alama wrote:
> This works for making a standalone executable that can exectute foo
> programs specified on the command line, but doesn't work for a REPL. The
> difficulty seems to be the `namespace-require` part of `run-repl`, defined
> like this:
>
> ````
> (define (run-repl)
> (parameterize ([current-namespace (make-base-empty-namespace)])
> (namespace-require 'foo/expander)
> (read-eval-print-loop)))
> ````
>
> When I invoke the executable with no arguments, `run-repl` gets called. But
> this leads to:
>
> ````
> standard-module-name-resolver: collection not found
> for module path: racket/base/lang/reader
> collection: "racket/base/lang"
> ````

The problem is that `racket/base` is declared only in the original
namespace. When you create a fresh namespace, then it gets a fresh
registry, and `racket/base` is not in that registry.

Really, that goes for `foo/expander`. It's not so much that you want to
refer to `racket/base` as `foo/expander`. While `++lang foo` should
make `foo/expander` be in the original namespace's registry (assuming
that the `foo` language refers to it), it won't be in the fresh
namespace's registry.

The solution is to attach the module declaration from the original
namespace to the new namespace.

Here's an example to demonstrate, assuming that the "foo" directory is
installed as a package. The `foo` language here supports just literals
and addition.

;; ----------------------------------------
;; foo/main.rkt
#lang racket/base

(provide #%app
#%datum
#%module-begin
#%top-interaction
+)

(module reader syntax/module-reader
foo)

;; ----------------------------------------
;; repl.rkt
#lang racket/base
(require racket/cmdline
;; Ensure that `foo` is here to attach
(only-in foo))

(command-line
#:args
([file #f])
(cond
[(not file)
(define ns (make-base-empty-namespace))
(namespace-attach-module (current-namespace) 'foo ns)
(current-namespace ns)
(namespace-require 'foo)
(read-eval-print-loop)]
[else
(dynamic-require file #f)]))

;; ----------------------------------------
;; Command line
raco exe ++lang foo repl.rkt
raco dist repl-dist repl

[Jesse Alama]

Thanks! This helps. I didn't know I was sailing into deep namespace waters. But another error has reared its head: when, in the REPL, I evaluate an expression in the foo language, I get

standard-module-name-resolver: collection not found

  for module path: racket/match/gen-match

  collection: "racket/match"

  in collection directories:

   /tmp/build/dist/lib/plt/foo/collects/

  context...:

   show-collection-err

   standard-module-name-resolver

   module-path-index-resolve5

   module-path-index-resolve5

   namespace-module-use->module+linklet-instances144

   for-loop

   temp37_0

   for-loop

   run-module-instance!125

   for-loop

   loop

   thunk

   dynamic-wind

   loop

   expand-capturing-lifts

   loop

racket/match is required in (sticking with your filenames) repl.rkt, as well as main.rkt and (not one of your filenames, but essentially just a source of macros that are in turn require'd in main.rkt) expander.rkt. I've added ++lib racket/match to raco exe, to no avail.

If I copy all standard Racket collects into the distribution (which makes racket/match in some sense "available"), I get a similar error (possibly the same error, essentially, just with a bit of more information added):

no module instance found: #<resolved-module-path:"/tmp/build/dist/lib/plt/foo/collects/racket/match/runtime.rkt"> 1

  context...:

   do-error

   namespace->module-instance70

   namespace-module-use->module+linklet-instances144

   for-loop

   temp37_0

   for-loop

   run-module-instance!125

   for-loop

   loop

   thunk

   dynamic-wind

   loop

   expand-capturing-lifts

   loop

   /Users/pltbuild/build/plt-release-64/bundle/racket/collects/racket/repl.rkt:11:26

   call-with-values

FWIW, the "1" there way at the end of the error message presumably comes from what I'm trying to evaluate in the REPL:

  (assignment "a" 1)

[Jesse Alama]

Thanks for helping out! It turns out that tip (1) doesn't make a difference, though, interestingly, (2) did, though it still doesn't quite get me over the goal line: the macros defined in expander.rkt seem to be undefined. It seems that the `compile` doesn't really have any effect on the namespace.

[Matthew Flatt]

At Thu, 26 Sep 2019 23:06:28 -0700 (PDT), Jesse Alama wrote:
> On Thursday, September 26, 2019 at 3:37:43 PM UTC+2, Matthew Flatt wrote:
> Thanks! This helps. I didn't know I was sailing into deep namespace waters.
> But another error has reared its head: when, in the REPL, I evaluate an
> expression in the foo language, I get
>
> standard-module-name-resolver: collection not found
> for module path: racket/match/gen-match
> collection: "racket/match"
> in collection directories:
> /tmp/build/dist/lib/plt/foo/collects/

It looks like `lazy-require` as used by `racket/match` isn't compatible
with creating a new namespace and attaching `racket/match` to that
namespace. More generally, `define-runtime-module-path-index` doesn't
cooperate with `namespace-attach-module`. Offhand, I don't know how
this might be fixed.

For your application, I wonder whether you need to create a new
namespace at all. The namespace should start out empty, so just
`namespace-require` should set it up ok. Do you need to support
multiple REPLs or ohterwise have multiple namespaces?

[Jesse Alama]

On Friday, September 27, 2019 at 2:25:17 PM UTC+2, Matthew Flatt wrote:

It looks like `lazy-require` as used by `racket/match` isn't compatible
with creating a new namespace and attaching `racket/match` to that
namespace. More generally, `define-runtime-module-path-index` doesn't
cooperate with `namespace-attach-module`. Offhand, I don't know how
this might be fixed.

For your application, I wonder whether you need to create a new
namespace at all. The namespace should start out empty, so just
`namespace-require` should set it up ok. Do you need to support
multiple REPLs or ohterwise have multiple namespaces?

I don't currently need multiple REPLs or namespaces (I think). Using your question and your wondering whether new namespaces are needed as clues, I trimmed the "start a REPL" function down to this:

(define (run-repl)

  (current-read-interaction foo:read-syntax-for-repl)

  (read-eval-print-loop))

But with this in place I get, in the REPL, trying to evaluate the string "$a := 1" (that's the non-S-expression syntax for a simple assignment):

unknown:1:0: #%top-interaction: unbound identifier;

 also, no #%app syntax transformer is bound

  at: #%top-interaction

  in: (#%top-interaction riposte-repl (normal-assignment "a" (expression 1)))

Somehow, #%app and #%top-interaction got lost.

[Matthew Flatt]

Do you still have `(namespace-require 'foo/expander)` somewhere? That
is still needed to add bindings to the namespace.

That is, all of the modules in the executable will be declared in the
namespace's registry, but no bindings will have been `require`d into
the namespace.

[Jesse Alama]

D'oh. That did the trick. Here's what I've got now:

(define (run-repl)

  (current-read-interaction foo:read-syntax-for-repl)

  (namespace-require 'foo/expander)

  (read-eval-print-loop)) 

That's it. Works like a charm. Thanks! I seemed to have been overthinking things; I thought that I needed a fresh namespace.